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FOREWORD 


1.  AFOTEC  Pamphlet  400-1  provides  logistics  personnel  with  a  base  on  which  to  build  knowledge 
of  concepts  and  procedures  pertaining  to  operational  suitability  test  and  evaluation.  It  is  intended 
to  serve  as  a  "how  to"  guide  for  HQ  AFOTEC/LG  staff*  AFOTEC  detachments  and  test  teams  and 
other  agencies  involved  in  operational  test  and  evaluation  (OT&E)  may  also  find  it  useful.  The 
pamphlet  is  divided  into  three  parts  as  follows: 

a.  Part  I  gives  an  overview  of  the  pamphlet  and  operational  suitability  test  and  evaluation. 

b.  Part  II  deals  with  the  key  activities  of  operational  suitability  test  and  evaluation,  i.e., 
test  concept  development,  operational  assessments,  test  execution,  and  test  reporting.  Included  is 
a  discussion  of  participating  organizations  and  their  relationships. 

c.  Part  III  provides  detailed  discussions  on  key  operational  suitability  evaluation  areas,  i.e.. 
availability,  reliability,  maintainability,  software  evaluation,  and  integrated  diagnostics. 
Attachments  provide  additional  discussions  of  special  operational  suitability  areas/issues  such  as 
dormant  reliability,  nuclear  hardness  maintenance/hardness  surveillance  (HM/HS),  and  considera¬ 
tions  of  availability  evaluations  by  system  type. 

2.  Most  of  the  chapters  include  a  discussion  of  lessons  learned  relative  to  the  chapter  topic.  These 
lessons  are  not  meant  to  be  all  inclusive.  The  dynamics  of  the  OT&E  business  will  constantly 
require  judgment  on  the  part  of  operational  suitability/logistics  personnel  when  applying  these 
lessons  to  their  assigned  program/mission. 

3.  As  a  final  note,  we  realize  that  publication  of  this  pamphlet  precedes  several  key  Air  Force 
regulations  (AFR)  (AFR  57-1  and  800  series).  This  impact  is  reduced  if  the  following  points  are 
kept  in  mind: 

a.  The  cost  and  operational  effectiveness  analysis  (COEA)  is  a  new  key  acquisition  document. 
Experience  with  it  will  be  incorporated  in  future  updates  to  this  pamphlet.  Also  experience  with 
the  evolutionary  requirements  process,  exit  criteria,  and  system  maturity  matrices  is  not  covered 
in  this  issuance  of  the  pamphlet. 

b.  A  new  procedural  guide  to  replace  LGOI  400-1,  Qualitative  Maintainability  and  Logistics 
Supportability  Questionnaires,  is  planned. 

c.  Results  from  AFOTEC  general  support  contract  studies  on  integrated  diagnostics  are  not 
complete  and  so  are  not  included  in  the  chapter  on  integrated  diagnostics. 

4.  Requests  for  copies  of,  changes  to,  or  questions  concerning  this  document  should  be  made  to: 

HQ  AFOTEC/LG3. 

Kirtland  AFB,  New  Mexico  87117-7001 


Supersedes  AFOTECP  400-1,  29  February  1984 
No.  of  Printed  Pages:  155 
OPR:  HQ  AFOTEC/LG  (Mr  D.  Young) 
Approved  by:  Col  Hubbard 
Distribution:  F;  X:  DTIC-FDAC 
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Part  I 
OVERVIEW 
Chapter  1 
INTRODUCTION 


1-1.  Objectives: 

a.  AFOTECP  400-1  is  a  guide  for  plan¬ 
ning,  executing,  and  reporting  an  Air  Force 
operational  suitability  test  and  evaluation 
(T&E). 

b.  Some  uses  of  the  pamphlet  are: 

(1)  A  training  aid  for  logistics  personnel 
unfamiliar  with  concepts  and  procedures  con¬ 
cerning  operational  suitability  test  and  evalu¬ 
ation. 

(2)  A  guide  for  AFOTEC  logistics  staff  in 
preparing  and  evaluating  test  plans  and  data 
management  and  analysis  plans  (DMAP). 

(3)  A  guide  for  operating  command  per¬ 
sonnel  preparing  operational  suitability 
portion  of  test  plans. 

(4)  A  guide  for  test  team  personnel  ex¬ 
ecuting  and  reporting  operational  suitability 
test  and  evaluation. 

1-2.  Organization: 

a.  AFOTECP  400-1  provides  logistics  and 
software  evaluation  managers,  analysts,  and 
test  team  members  with  "how  to"  information 
helpful  in  planning,  executing,  and  reporting 
operational  suitability  evaluations  of  systems. 
Part  one  contains  an  overview  of  the  opera¬ 
tional  suitability  T&E  process. 

b.  Part  two  focuses  on  how  operational 
suitability  T&E  is  planned  and  executed. 
The  description  of  test  planning  progresses 
from  general  to  specific  cases,  and  the  discus¬ 
sion  of  test  execution  progresses  from  prepa¬ 


ration  through  reporting. 

c.  Part  three  examines  key  operational 
suitability  areas  and  the  relationship  between 
the  three  basic  quantitative  system  param¬ 
eters  availability,  reliability,  and  maintain¬ 
ability.  Simulation  models  available  to  assist 
in  extending  test  results  are  identified  and 
briefly  described. 

d.  The  attachments  expand  on  specific 
areas/issues  of  operational  suitability  test  and 
evaluation  such  as  document  reliability, 
nuclear  HM/HS,  and  availability  evaluation 
of  specific  systems. 

1-3.  Principal  Sources  of  System  Infor¬ 
mation.  The  ability  to  perform  an  opera¬ 
tional  test  and  evaluation  (OT&E)  which  is 
responsive  to  program  decision  needs  is 
dependent  on  the  availability  of  system- 
specific  information.  Much  of  the  informa¬ 
tion  is  available  from  the  implementing  and 
using  commands  and  DOD.  For  major  sys¬ 
tems,  a  series  of  documents  provide  this 
information  as  described  in  attachment  5. 
Use  of  these  documents  in  test  planning  and 
execution  is  explained  in  part  two.  In  addi¬ 
tion,  major  programs  should  have  additional 
documentation  such  as  a  Program  Manage¬ 
ment  Plan  (PMP),  Reliability  and  Main¬ 
tainability  Management  Plan  (RMMP),  and 
a  Life  Cycle  Cost  Management  Plan.  Less- 
than-major  systems  generally  have  fewer 
documents  available. 
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Chapter  2 

OPERATIONAL  SUITABILITY  TEST  AND  EVALUATION 


2-1.  Rationale  for  Conducting  Opera¬ 
tional  Suitability  OT&E.  The  objective  of 
operational  suitability  test  and  evaluation  is 
to  ensure  that  emerging  systems  can  be 
operated  and  maintained  in  field  conditions. 
Operational  suitability  test  and  evaluation 
often  results  in  changes  in  system  design 
which  result  in  improved  availability  and 
significant  savings  in  spares,  fuel,  manpower, 
and  other  scarce  logistics  resources. 

2-2.  Suitability  Test  and  Evaluation. 
Operational  suitability  is  the  degree  to  which 
a  system  can  be  satisfactorily  placed  in  field 
use,  with  consideration  given  to  availability, 
compatibility,  transportability,  inter¬ 
operability,  reliability,  wartime  usage  rates, 
maintainability,  safety,  human  factors,  man¬ 
power  supportability,  logistics  supportability, 
and  training  requirements.  While  logistics 
personnel  are  generally  responsible  for  the 
operational  suitability  evaluation,  the  division 
of  responsibility  for  several  of  the  suitability 
areas  (or  elements)  defined  above  is  not 
clear.  For  example,  safety,  human  factors, 
compatibility,  and  interoperability  apply  to 
both  operational  suitability  and  operational 
effectiveness.  These  areas  may  be  addressed 
under  suitability  and  effectiveness,  by  nei¬ 
ther,  or  by  one  or  the  other  dependent  on  the 
unique  aspects  of  the  system  being  evaluated. 
Figure  2-1  portrays  the  relationship  among 
the  areas  of  operational  suitability.  Every 
suitability  test  and  evaluation  should: 

a.  Measure  the  hardware  and  software 
suitability  performance  of  the  system. 

b.  Document  and  track  deficiencies  and 
proposed  enhancements  using  a  service 
reporting  system. 

c.  Evaluate  corrective  actions  for  identi¬ 
fied  deficiencies  to  ensure  they  adequately 
resolve  the  original  problem  without  causing 
new  problems. 

d.  Estimate  the  mature  operational  suita¬ 
bility  of  the  system,  primarily  through  model¬ 
ing  and  simulation. 

e.  Compare  the  estimated  mature  perform¬ 
ance  with  the  user  requirements. 

2-3.  System  Operating  Phase  Analysis. 
Most  systems  of  interest  are  designed  to 
operate  in  three  distinct  phases:  peacetime 
operations,  increased  readiness  postures,  and 
actual  wartime  mission  execution.  The 
system  operational  requirements  for  each  of 
these  phases  may  be  different,  as  are  the 


primary  measures  of  system  suitability  per¬ 
formance.  The  environment  during  OT&E 
comes  closest  to  emulating  the  peacetime- 
phase  but,  because  of  the  many  constraints 
involved,  is  often  not  truly  representative  of 
even  peacetime  operational  conditions. 

2-4.  Constraints  on  Operational  Suit¬ 
ability  Testing: 

a.  Operational  suitability  test  and  evalua¬ 

tion  is  conducted  throughout  the  acquisition 
process.  During  the  earliest  stages  of  a 
system’s  evolution,  it  is  obvious  any  evalua¬ 
tion  must  be  accomplished  without  the  bene¬ 
fit  of  the  system — it  has  not  yet  left  the 
drawing  boards.  Operational  suitability  tests 
are  frequently  conducted  with  the  following 
limitations:  limited  test  assets,  prototype 

equipment,  and  immature  or  nonexistent 
logistics  elements  such  as  technical  orders, 
supply  support,  and  support  equipment. 

b.  An  obvious  limitation  is  the  inability  to 
test  the  system  in  a  wartime  environment. 
A  primary  operational  test  objective  is  to 
execute  the  test  in  as  realistic  an  environ¬ 
ment  as  possible.  In  some  instances,  the 
system  is  deployed  to  a  potential  operational 
site  for  testing.  However,  considerations 
such  as  test  schedule,  funds  limitations,  and 
system  immaturity  often  preclude  testing  at 
other  than  a  test  range.  Consequently, 
system  analysis  modeling  is  frequently  used. 
Models  provide  the  capability  to  tackle  issues 
such  as  a  system’s  logistics  characteristics 
during  war,  to  give  significance  and  utility  to 
test  results  beyond  the  narrow  conditions  of 
the  test,  and  to  develop  a  reasonably  ac¬ 
curate  measure  of  the  system’s  availability 
and  mission  reliability.  Modeling  techniques 
are  discussed  in  part  three  of  this  pamphlet. 

2-5.  Lessons  Learned.  Lessons  learned 
from  past  OT&E  programs  are  a  valuable 
source  of  information  for  planning  and  exe¬ 
cuting  suitability  assessments.  Most  chapters 
of  this  pamphlet  conclude  with  a  discussion 
of  lessons  learned. 

2-6.  USAF  OT&E  Lessons  Learned  Pro¬ 
gram.  The  objective  of  the  formal  OT&E 
lessons  learned  program,  administered  by  HQ 
AFOTEC/XPX,  is  to  ensure  future  efforts 
benefit  from  past  experience.  Lessons 
learned  reports  which  document  technical, 
management,  and  operations  security  (OPSEC) 
are  contained  in  the  lessons  learned  file  of 


COMPATIBILITY 


.  Areas  of  Operational  Suitability 
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the  USAF  OT&E  data  bank.  (For  more  in-  800-1.  OT&E  Lessons  Learned  Program.) 

formation,  refer  to  AFR  55-43  and  AFOTECR 
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3-1 


3-1.  Introduction.  This  chapter  outlines 
the  common  (and  generally  qualitative) 
considerations  of  operational  suitability  and 
examines  some  factors  that  most  commonly 
affect  these  considerations.  Chapter  3  should 
be  used  to  supplement  chapter  7  and  part 
three.  The  following  considerations  are 
addressed  herein: 

a.  Qualitative  maintainability. 

b.  Logistics  supportability. 

c.  Support  equipment. 

d.  Supply  provisioning. 

e.  Technical  data. 

f.  Packaging,  handling,  and  transporta¬ 
tion. 

g.  Facilities. 

h.  Maintenance  training. 

i.  Maintenance  planning. 

j.  Software. 

k.  Other  considerations. 

Questionnaires  used  in  the  assessment  of 
some  of  these  areas  can  be  found  in  LGOI 
400-1,  Qualitative  Maintainability  and  Logis¬ 
tics  Supportability  Questionnaires. 

3-2.  Qualitative  Maintainability: 

a.  Maintenance  Actions.  The  planner 
should  consider  qualitative  maintainability 
(i.e.,  accessibility,  serviceability,  ease  or  diffi¬ 
culty  of  maintenance,  safety,  and  human 
factors)  associated  with  maintenance  actions, 
as  well  as  quantitative  aspects  of  maintain¬ 
ability.  General  considerations  include: 

(1)  Are  the  components  easily  removed? 

(2)  What  effect  will  adverse  environment¬ 
al  conditions  such  as  cold  weather  or  CBW 
have  on  the  technicians’  ability  to  perform 
maintenance  tasks? 

(3)  Since  maintainability  is  also  con¬ 
cerned  with  servicing,  inspecting,  trouble¬ 
shooting,  repairing,  removal,  and  replacement 
tasks,  at  what  level  (e.g.,  flight  line,  shop, 
depot)  can  specific  maintenance  actions  be 
accomplished? 

b.  Maintainability  Assessment.  An¬ 
swers  to  these  questions  will  affect  the  quan¬ 
tity,  skill  level,  and  specialty  code  of  person¬ 
nel  and  the  test  equipment  required  to 
maintain  a  system.  The  importance  of  this 
assessment  cannot  be  overemphasized.  His¬ 
torically,  over  50  percent  of  the  OT&E  serv¬ 
ice  reports  result  from  qualitative  maintain¬ 
ability  assessment. 

c.  Test  Methodology.  Experienced 
maintenance  technicians  should  conduct  the 
qualitative  assessment  of  maintainability. 


AFOTEC  has  developed  a  procedural  guide 
(LGOI)  which  contains  a  general  checklist  for 
such  an  assessment.  ITie  checklist  is  de¬ 
signed  to  be  modified  by  headquarters  and 
test  team  personnel  based  on  specific  pro¬ 
gram  requirements. 

d.  Data  Requirements.  An  assessment 
form  or  questionnaire  will  normally  be  used 
to  gather  data  for  the  qualitative  assessment. 
These  data  are  often  used  as  supporting  data 
for  the  quantitative  measures.  Where  feas¬ 
ible,  use  of  video  recording  equipment  should 
also  be  considered  to  gather  information  for 
both  qualitative  and  quantitative  assessments 
and  evaluations. 

e.  Assessment.  For  the  qualitative 
assessment  of  maintainability,  subjective 
judgments  will  normally  be  used.  However, 
these  judgments  can  often  be  supported  with 
the  quantitative  tools  discussed  under  the 
reliability  and  maintainability  evaluations. 
The  following  are  general  points  to  be  consid¬ 
ered: 

(1)  Time-consuming  tasks  which  affect 
turnaround  times  or  downtime. 

(2)  Tasks  for  which  a  special  tool  will 
solve  a  maintainability  problem. 

(3)  Scope  and  frequency  of  inspections  in 
areas  with  poor  accessibility  or  location. 

(4)  Human  factors  considerations  (e.g., 
weight,  handles,  height  above  ground  level, 
etc.). 

(5)  Effects  of  climatic  extremes  on  the 
ability  of  maintenance  personnel  to  perform 
tasks. 

(6)  Ease  of  removal,  replacement,  repair, 
fault  isolation,  inspecting,  lubrication,  and 
servicing  of  the  weapon  system. 

(7)  Minimum  crew  size  requirements, 
when  applicable. 

(8)  Special  handling  and  protective  equip¬ 
ment  (particularly  chemical,  biological,  and 
radiological  (CBR)). 

(9)  Provisions  for  nondestructive  inspec¬ 
tions. 

(10)  Provisions  for  crash  recovery. 

(11)  Unsatisfactory  or  unsafe  maintenance 
procedures  dictated  or  promoted  by  com¬ 
ponent,  hardware,  or  installation  design. 

3-3.  Logistics  Supportability.  Logistics 
supportability  assessment  addresses  the 
various  elements  of  integrated  logistics  sup¬ 
port  (ILS),  e.g.,  technical  data,  facilities,  and 
maintenance  training.  The  ability  to  assess 
ILS  elements  depends  on  the  test  environ- 
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ment.  For  example  during  IOT&E,  support 
equipment  and  technical  data  may  not  be 
available  or  representative  of  the  operational 
items.  Further,  the  contractor  typically 
provides  supply  support.  Test  plans  should 
be  written  to  give  the  test  team  latitude  to 
document  any  logistics  supp^rtability  observa¬ 
tions  they  may  be  able  to  make  within  the 
constraints  of  the  test. 

3-4.  Support  Equipment  (SE).  For  OT&E 
purposes,  SE  can  be  divided  into  two  class¬ 
es — major  and  nonmajor.  In  reality,  which 
kind  of  evaluation  to  perform  on  a  given 
piece  of  SE  will  not  be  an  either/or  decision 
but  will  be  a  matter  of  degree.  The  planner 
should  consider  such  factors  as  unit  cost, 
degree  of  risk,  projected  utilization  rates, 
maintenance  requirements,  and  complexity  in 
determining  the  extent  of  evaluation  re¬ 
quired.  Major  equipment  usually  includes 
such  items  as  avionics  systems  automatic  test 
stations,  newly  designed  complex  SE,  and 
similar  equipment.  Nonmajor  equipment 
usually  includes  such  items  as  nonpowered 
SE,  hand  tools,  and  SE  that  is  already  in  the 
inventory.  The  selection  of  measures  of 
effectiveness  (MOE)  will  depend  on  the 
complexity  of  the  equipment  and  its  function. 
Normally,  only  a  few  major  types  of  equip¬ 
ment,  such  as  avionics  test  stations,  will 
have  quantitative  requirements  listed  in  the 
maintenance  concept  or  other  program  docu¬ 
mentation.  For  these  types  of  SE,  compre¬ 
hensive  reliability,  maintainability,  availabil¬ 
ity,  and  logistics  supportability  evaluations/ 
assessments  should  be  conducted  and  the 
MOEs  selected  accordingly. 

a.  General  Considerations.  Factors  to 
be  considered  in  developing  SE  objectives 
and  MOEs  are: 

(1)  Reliability. 

(2)  Maintainability. 

(3)  Availability. 

(4)  Suitability  for  mobility,  deployment, 
and  bare  base  operations,  as  appropriate. 

(5)  Ease  or  difficulty  of  operation. 

(6)  Effectiveness  in  performing  trouble¬ 
shooting  and  diagnostic  functions. 

(7)  Requirements  for  test,  measurement, 
and  diagnostic  equipment  (TMDE)  support. 

(8)  Human  factors,  such  as  operator- 
machine  interface,  ease  of  handling,  weight 
and  size  of  components,  etc. 

(9)  Susceptibility  to  damage,  contami¬ 
nation,  or  corrosion. 

(10)  Quantity  of  units  needed  versus 
authorized. 

(11)  Other  ILS  elements’  capability  to 
support  the  SE  (that  is,  technical  data, 


supply  support,  facilities,  etc.). 

( 12)  Safety. 

'13)  Compatibility. 

b.  Test  Methodology.  For  major  SE.  a 
comprehensive  test  methodology  which  is 
nearly  as  detailed  as  that  used  for  the  prime 
system  may  be  required. 

c.  Data  Requirements: 

( 1)  SE  may  have  one  or  more  of  the 
following  techniques  applied,  depending  on 
type  and  complexity  of  the  equipment: 

(a)  Data  collection  ( automated  or  manu¬ 
al)  either  using  maintenance  data  collection 
system  <MDCS),  system  effectiveness  data 
system  (SEDS),  or  a  system  patterned  after 
them  to  record  data  and  make  basic  MOE 
computations. 

(b)  A  detailed  log  book  on  the  SE  in 
which  predetermined  significant  data  are 
recorded  for  a  specified  period  of  time. 

ic)  Realistic  reliability  and  maintaina¬ 
bility  (R&M)  demonstrations  during  which 
significant  predetermined  data  are  recorded. 

(d)  Predetermined  data  on  an  evalua¬ 
tion  form/questionnaire  each  time  the  equip¬ 
ment  is  used  or  maintained. 

(2)  For  comprehensive  evaluations,  use  of 
standard  MDCS  or  SEDS  is  desirable.  If  a 
contractor  data  system  must  be  used,  it 
should  approximate  the  Air  Force  systems  as 
closely  as  practical.  Over-the-shoulder  data 
collection  or  having  the  contractor  fill  out  Air 
Force  forms  and  then  processing  them  at  an 
Air  Force  facility  are  alternatives  which 
should  be  explored  thoroughly  before  becom¬ 
ing  dependent  on  contractor  data. 

(3)  Data  collection  using  log  books  or  test 
team-developed  forms  will  require  manual 
processing  and  analysis. 

(4)  References: 

(a)  AFR  66-1,  Maintenance  Manage¬ 
ment  Policy. 

(b)  AFR  66-14,  The  US  Air  Force 
Equipment  Maintenance  Program. 

(c)  AFM  67-1,  Basic  Air  Force  Supply 
Procedures. 

(d)  AFLCR  67-2,  USAF  Equipment 
Allowance  System. 

(e)  AFR  800-12,  Acquisition  of  Support 
Equipment. 

c.  Evaluation.  Identify  SE  deficiencies 
and  areas  requiring  enhancement  or  optimi¬ 
zation.  Such  an  evaluation  will  normally 
include  a  combination  of  quantitative  and 
qualitative  techniques. 

d.  Evaluation  Criteria.  Evaluation 
criteria  will  normally  only  be  developed  for 
those  MOEs  for  which  operational  require¬ 
ments  are  stated  in  the  program  documenta¬ 
tion. 
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3-5.  Supply  Support: 

a.  Operational  Readiness.  Maintain¬ 
ing  operational  readiness  under  diverse 
conditions  of  military  use  depends  directly  on 
the  right  supplies  being  available  at  the  time 
and  place  they  are  needed.  During  an 
OT&E,  supply  support  objectives  can  be 
divided  into  five  basic  categories: 

(1)  Providing  consumption,  failure,  and 
other  test  data  to  the  provisioner  in  order  to 
update  provisioning  factors. 

(2)  Comparing  test  data  with  the  provi¬ 
sioning  factors  to  identify  items  which  appear 
to  be  underprovisioned  or  overprovisioned. 

(3)  Reviewing  initial  spares  support  lists 
(ISSL),  bench  stock,  war  readiness  spares  kit 
(WRSK),  or  mission  support  kit  (MSK)  list¬ 
ings  and  comparing  them  with  test  results  to 
identify  items  which  appear  to  be  under  or 
in  excess  of  expected  requirements. 

(4)  Measuring  the  performance  of  the 
supply  support  system  and  identifying  defi¬ 
cient  areas. 

(5)  Reviewing  level  of  repair  decisions. 

b.  OT&E  Supply  Support.  Frequent¬ 
ly,  a  contractor  provides  OT&E  supply  sup¬ 
port.  Review  supply  planning  decisions  to 
compare  them  with  test  experience  to  date. 
This  is  an  ongoing  process  and  should  con¬ 
tinue  throughout  OT&E.  When  a  test  site  is 
located  at  an  operational  location  and  con¬ 
tractor  support  has  ended,  standard  Air 
Force  supply  support  may  begin.  Requisition 
fill  rates,  partially  mission  capable  supply 
(PMCS),  not  mission  capable  supply  (NMCS) 
rates,  and  other  supply  indices  may  be 
measured.  The  prime  supply  MOEs  are 
those  that  affect  the  availability  of  the  weap¬ 
on  system  (e.g.,  NMCS  rates). 

c.  General  Considerations.  The  plan¬ 
ner  should  consider  the  following  factors  in 
developing  supply  support  objectives  and 
MOEs: 

(1)  Component  reliability. 

(2)  Component  criticality. 

(3)  Effects  of  supply  support  on  availa¬ 
bility  (partially  mission  capable,  both  (main¬ 
tenance  and  supply)  (PMCB)  and  not  mission 
capable,  both  (maintenance  and  supply) 
(NMCB)). 

(4)  Not  repairable  this  station  (NRTS) 
rates  and  their  causes. 

(5)  Condemnation  rates. 

(6)  Bench  check  serviceable  rates  and 
causes. 

(7)  Cannot  duplicate  (CND)  rates  and 
causes. 

(8)  Cannibalization  rates. 

(9)  Mean  time  between  demands  (MTBD). 

(10)  ISSL,  WRSK,  MSK,  and  bench  stock 
listing  adequacy. 


(11)  Delayed  repair  of  components  a%vait- 
ing  parts. 

(12)  Average  component  repair  days 
(indicates  the  average  number  of  days  it 
takes  to  repair  an  item,  excluding  time 
awaiting  parts;. 

(13)  Inadequacies  in  technical  data  or 
SE  which  impact  supply  support. 

(14)  Source,  maintenance,  and  recov¬ 
erability  (SMR)  coding. 

d.  Test  Methodology: 

(1)  Select  the  more  critical  supply  items 
for  evaluation.  The  planner  should  consider 
factors  such  as  complexity,  cost,  criticality, 
and  failure  rates  in  selecting  these  candi¬ 
dates. 

(2)  To  evaluate  the  adequacy  of  the  total 
provisioning  process,  a  predetermined  number 
of  components  can  be  selected  randomly  and 
evaluated.  These  data  can  then  be  statisti¬ 
cally  treated  to  make  conclusions  about  the 
adequacy  of  the  provisioning  process  for  the 
population  as  a  whole. 

e.  Data  Requirements: 

(1)  The  reliability  and  availability  objec¬ 
tives  plus  the  data  from  the  provisioning 
agency  and  from  the  supply  tracking  system 
being  used  for  the  test  normally  satisfy  the 
data  requirements.  If  the  data  for  this 
requirement  must  come  from  the  contractor. 
test  planner  must  ensure  the  data  item 
descriptions  (DID)  from  which  the  data  are 
obtained  are  included  in  the  full-scale  devel¬ 
opment  (FSD)  contract. 

(2)  Other  required  data  will  include  such 
items  as  the  ISSL,  WRSK.  MSK,  and  bench 
stock  listings  and  logistics  support  analysis 
records  (LSAR).  The  provisioning  activity 
or  the  prime  Air  Logistics  Center  (ALC) 
normally  develops  and  provides  these  data. 

(3)  References: 

(a)  AFR  55-43.  Management  of  Opera¬ 
tional  Test  and  Evaluation. 

(b)  AFR  65-110,  Aerospace  Vehicle  and 
Equipment  Inventory,  Status,  and  Utilization 
Reporting  System  (AVISURS). 

(c)  AFR  66-1,  Maintenance  Manage¬ 
ment  Policy. 

(d)  AJFR  66-45,  Joint  Regulation  Gov¬ 
erning  the  Use  and  Application  of  Uniform 
Source,  Maintenance,  and  Recoverability 
Codes. 

(e)  AFM  67-1,  Basic  Air  Force  Supply 
Procedures. 

(f)  MIL-STD  1388,  Logistics  Support 
Analysis. 

f.  Evaluation.  Normally  the  evaluation 
of  supply  support  should  include: 

( 1)  Measuring  key  supply  parameters  and 
evaluating  them  against  criteria,  when  stated 
in  the  user  requirement  document. 
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(2)  Estimating  the  supply  system's  ability 
to  support  the  system  at  required  levels. 

(3)  Identifying  the  components  that  are 
underprovisioned. 

(4)  Reviewing  and  assessing  the  ade¬ 
quacy  of  the  WRSK,  ISSL,  MSK,  bench  stock 
listings,  and  LSARs  to  support  mission  re¬ 
quirements. 

(5)  Reviewing  and  assessing  the  repair 
level  analysis  (RLA)  and  the  resulting  SMR 
codes.  Make  recommendations  based  on  test 
experience. 

(6)  Conducting  a  random  statistical 
sample  of  provisioned  items  to  draw  conclu¬ 
sions  about  overall  provisioning  adequacy. 

(7)  Supplying  the  required  test  data  to 
the  provisioners. 

g.  Evaluation  Criteria: 

(1)  As  a  rale,  quantitative  evaluation 
criteria  are  appropriate  for  supply  only 
during  follow-on  OT&E  (FOT&E).  Until 
then,  the  supply  support  system  is  either  the 
responsibility  of  the  contractor  or  is  in  the 
initial  buildup  stage  for  organic  Air  Force 
support. 

( 2)  The  planner  can  apply  primary  evalu¬ 
ation  criteria  such  as  PMCB  and  NMCB. 

(a)  Key  measurements  of  reliability 
(mean  time  between  maintenance  (MTBM), 
mean  time  between  demand  (MTBD),  etc.) 
and  base-level  repair  capability  (NRTS  rates, 
etc.)  at  the  component  level  can  be  compared 
to  those  used  in  the  provisioning  process. 

(b)  However,  missing  the  mark  on  these 
comparisons  will  not  in  itself  mean  an  item 
has  been  underprovisioned  or  overprovi¬ 
sioned.  It  will  simply  identify  a  candidate 
for  further  evaluation.  Such  factors  as 
anticipated  reliability  improvements,  im¬ 
proved  technical  data,  and  increased  base- 
level  repair  capability  must  then  be  con¬ 
sidered  before  determining  that  a  component 
is  underprovisioned  or  overprovisioned. 

(3)  Another  useful  criterion  for  iden¬ 
tifying  potential  supply  problems  is  to  iden¬ 
tify  the  top  PMCS/NMCS  contributors  and 
the  top  cannibalization  items. 

3-6.  Technical  Data.  Technical  data  are 
the  link  between  personnel  and  equipment. 
Traditionally,  they  have  been  paper  products, 
but  the  current  USAF  trend  is  toward  auto¬ 
mation,  i.e.,  digital  technical  data.  Adequacy, 
usability,  completeness,  correctness,  and 
understandability  of  technical  data  should  be 
assessed.  The  assessment  of  technical  data 
is  a  subjective  process. 

a.  General  Considerations.  The  follow¬ 
ing  factors  should  be  considered  in  developing 
objectives  and  MOEs: 

(1)  Adequacy  of  the  contractor’s  plan  to 


provide  validated  manuals  in  time  to  meet 
the  Air  Force  verification  schedule. 

(2)  Adequacy  of  the  Air  Force  verifica¬ 
tion  plan  to  provide  formal  publications  in 
the  most  efficient  and  cost-efficient  and  cost- 
effective  manner. 

(3)  Provisions  for  checklist  format  where 
sequential  steps  and  tasks  to  be  accomplished 
make  them  appropriate. 

(4)  Adequate  notes,  cautions,  and  warn¬ 
ings  for  personnel  safety  and  protection  of 
equipment. 

(5)  Identification  of  hardness  critical 
processes  and  hardness  critical  items. 

(6)  Usefulness  of  the  form  of  technical 
data  (automated,  checklists,  pocket  size,  etc.). 

(7)  Adequacy  of  illustrations. 

(8)  References  to  special  tools  and  test 
equipment. 

(9)  Utility  of  table  of  contents  and  index. 

(10)  Need  to  refer  to  other  TOs  and 
manuals  to  complete  tasks. 

(11)  Consistency  in  installation,  hard¬ 
ware,  and  safety  provisions  between  related 
manuals. 

(12)  Adequacy  of  troubleshooting  proce¬ 
dures. 

(13)  Adequacy  of  illustrated  parts  break¬ 
down. 

(14)  Readability  of  the  TO. 

b.  Test  Methodology.  The  test  team 
should  participate  and  provide  inputs  through 
the  program  office  into  the  review  of  the 
technical  data  specifications,  table-top  reviews 
of  technical  data,  and  the  contractor’s  valida¬ 
tion  effort.  The  team  should  evaluate  the 
technical  data  during  its  day-to-day  use  and 
participate  and  provide  inputs  to  the  program 
office  during  in-process  reviews  to  ensure  all 
previously  found  discrepancies  are  corrected. 

c.  Data  Requirements: 

(1)  Data  requirements  usually  include 
AFTO  Form  158  for  preliminary  technical 
orders  (TO),  AFTO  Forms  22  (Technical 
Order  System  Publication  Improvement 
Report  and  Reply),  formal  publications,  or 
AFTO  Forms  27  (Technical  Order  System 
Publication  Change  Request  (PCR))  (sub¬ 
mitted  LAW  TO  00-5-1,  Air  Force  Technical 
Order  System)  for  preliminary  publications. 
These  are  the  deficiency  reports  for  technical 
data.  Also  needed  will  be  the  proposed  and 
actual  delivery  schedules  of  the  publications 
and  the  validation  and  verification  plans  and 
schedules.  Properly  designed  questionnaires 
may  also  prove  helpful.  The  planner  will 
require  the  minutes  and  worksheets  from  any 
specification  reviews,  prepublication  reviews, 
prepublication  reviews,  or  other  such  meet¬ 
ings. 

(2)  References: 
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(a)  AFR  8-2,  Air  Force  Technical  Order 
System. 

(b)  AFR  310-1,  Management  of  Contrac¬ 
tor  Data. 

(c)  AFSCM  310-2,  Technical  Publica¬ 
tions  Acquisition  Management. 

(d)  TO  00-5-1,  Air  Force  Technical 
Order  System. 

(e)  TO  00-5-2,  Air  Force  Technical 
Order  Distribution  System. 

(f)  TO  00-5-15,  Air  Force  Time  Com¬ 
pliance  Technical  Order  System. 

(g)  MIL-L  8031,  List  of  Applicable 
Publications  (LOAP). 

<L  Assessment.  The  assessment  of  tech¬ 
nical  data  should  determine  the  adequacy  of 
the  technical  data  to  support  mission  require¬ 
ments,  assess  the  timeliness  of  the  delivery 
of  the  to  support  operational  requirements, 
and  identify  technical  data  deficiencies  with 
recommendations,  when  appropriate. 

e.  Assessment  Criteria.  Technical  data 
can  be  assessed  in  terms  of  the  adequacy, 
usability,  completeness,  correctness,  and 
understandability  of  the  data  for  their  in¬ 
tended  use.  This  is  a  qualitative  assessment 
requiring  judgmental  criteria.  Technical  data 
can  also  be  assessed  in  terms  of  the  delivery 
of  technical  data  versus  the  required  sched¬ 
ule. 

3-7.  Packaging,  Handling,  and  Trans¬ 
portation  (PHT).  PHT  assessment  involves 
an  effort  to  ensure  the  capability  to  trans¬ 
port,  preserve,  package,  and  handle  all  sys¬ 
tem,  equipment,  and  support  items.  Mission, 
design  specifications,  item  configuration, 
safety,  geographic  and  environmental  consid¬ 
erations,  or  packaging  and  preservation  con¬ 
cepts  may  dictate  requirements.  OT&E  inter¬ 
ests  range  from  the  adequacy  of  packing 
materials  and  techniques  and  preservation 
procedures  for  shipping  spare  parts  (particu¬ 
larly  cure-dated  and  fragile  items)  to  the 
transportability  of  an  entire  system,  including 
its  teardown  and  reassembly.  Such  items  as 
protection  from  weather  and  from  rough  han¬ 
dling  are  prime  considerations.  The  assess¬ 
ment  of  handling  and  transportation  equip¬ 
ment  is  normally  a  qualitative  process. 

a.  General  Considerations.  These 
factors  should  be  considered  in  developing 
specific  objectives  and  MOEs: 

(1)  Outsized  components  and  peculiar 
requirements  for  packaging,  crating,  han¬ 
dling,  or  special  precautions. 

(2)  Adequacy  of  provisions  for  timely 
deployment  and  redeployment. 

(3)  Provisions  for  handling  and  trans¬ 
portation  within  the  organization. 

(4)  Adequacy  of  shipping  containers. 
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(5)  Suitability  of  handling,  carrying  de¬ 
vices,  and  protective  devices  (dust,  shock, 
impact,  moisture). 

b.  Test  Methodology.  Packaging  special¬ 
ists  should  assist  in  the  evaluation  of  han¬ 
dling  and  transportation.  They  can  be  as¬ 
signed  to  the  test  team  on  a  TDY  basis  from 
the  ALC  which  has  the  prime  responsibility 
for  the  weapon  system.  The  following  meth¬ 
ods  of  assessment  should  normally  be  used: 

(1)  Review  the  contractor’s  and  ALC’s 
packaging  and  transportation  plans  for  ade¬ 
quacy  and  compliance  with  the  governing 
directives  and  specification;  evaluate  the 
equipment,  when  the  test  team  receives  it, 
using  a  test  team-developed  evaluation  form 
and  checklist;  and  identify  deficiencies  as  the 
equipment  is  used  on  a  day-to-day  basis. 
(When  suspected  deficiencies  are  identified, 
the  test  team  should  examine  and  compare 
the  equipment  to  applicable  directives  to 
determine  compliance  with,  or  adequacy  of, 
those  directives  or  specifications  listed  below.) 

(2)  When  using  either  the  MDCS  or 
SEDS,  the  test  team  should  review  main¬ 
tenance  data  for  incidents  of  when-discovered 
code  Y,  "upon  receipt  or  withdrawal  for 
supply  stocks,”  and  how-malfunction  code 
086,  "improper  handling."  The  team  should 
investigate  these  incidents  for  shortcomings 
with  handling  and  transportation  equipment. 

c.  Data  Requirements.  Data  require¬ 
ments  generally  include  contractor  and  depot 
packaging  and  transportation  plans,  test 
team  assessment  form  or  checklist,  MDCS 
and  SEDS  data  products,  and  applicable 
directives  and  specifications.  These  usually 
include  at  least  the  following: 

(1)  AFR  71-1,  Packaging  Management. 

(2)  AFR  71-4,  Preparing  Hazardous 
Materials  for  Military  Air  Shipment. 

(3)  AFR  71-9,  Air  Force  Packaging. 

(4)  AFR  80-18,  Department  of  Defense 
Engineering  for  Transportability. 

(5)  AFLCM  71-1,  AFLC  Packaging  and 
Materials  Handling  Policies  and  Procedures. 

(6)  MIL-STD  1367,  Packaging,  Handling, 
Storage,  and  Transportability  Program  Re¬ 
quirements  (For  Systems  and  Equipments). 

(7)  MIL-P  9024,  Packaging,  Handling, 
and  Transportability  in  System/Equipment 
Acquisition. 

d.  Assessment.  The  assessment  of  trans¬ 
portation  and  handling  equipment  should 
normally  include  the  factors  discussed  herein. 

e.  Assessment  Criteria.  Subjective  judg¬ 
ments  of  test  team  personnel,  identification 
of  damaged  components  or  equipment  attrib¬ 
utable  to  handling  and  transportation  equip¬ 
ment,  and  comparison  of  the  equipment  to 
applicable  directives  and  specifications  are 
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some  criteria  that  mav  be  used  in  assessing 

PHT. 

3-6.  Maintenance  Facilities.  Maintenance 
facilities  planning  is  based  on  engineering, 
operational,  and  maintenance  requirements. 
The  test  team  should  monitor  all  mainte¬ 
nance  activities  to  identify  any  requirements 
that  have  not  been  satisfied.  The  test  team 
should  use  AFM  86-2,  Standard  Facility 
Requirements,  as  a  guide.  Close  coordination 
with  the  prime  ALC  and  using  command  is 
important  in  this  assessment,  and  the  test 
team  should  review  all  applicable  facilities 
plans  against  test  experience  to  make  sure 
the  proper  factors  have  been  used  in  deter¬ 
mining  facilities  requirements.  The  LSA- 
012  report  may  be  a  useful  tool  in  deter¬ 
mining  the  various  operations  and  mainte¬ 
nance  (O&M)  functions  to  be  performed  in 
planned  facilities.  Such  items  as  heavy 
maintenance  docks,  work  areas,  storage 
requirements,  wash  rack,  test  cell  require¬ 
ments,  etc.,  are  of  interest. 

a.  General  Consideration.  These  fac¬ 
tors  should  be  considered  in  developing 
objectives  and  MOEs: 

(1)  Programmed  and  forecast  utilization 
rates. 

(2)  Number  of  systems  and  units  per 
squadron  or  wing. 

(3)  TMDE  authorizations  per  squadron 
or  wing. 

(4)  SE  authorizations  per  squadron  or 
wing. 

(5)  Clean-room  requirements. 

(6)  Wash  rack,  phase  inspection  dock,  fuel 
cell,  and  similar  requirements. 

(7)  Munitions  storage  requirements. 

(8)  Utilities  requirements. 

(9)  Security  requirements. 

(10)  Environmental  control  requirements. 

(11)  War  readiness  material  (WRM) 
storage  requirements. 

(12)  Manpower  authorizations. 

(13)  Forward  operating  location  and 
deployment  requirements. 

(14)  Hazardous  materials  handling  and 
disposal  requirements. 

b.  Test  Methodology: 

(1)  The  evaluation  of  facilities  may  cover 
site  activation  activities  by  working  with  the 
site  activation  task  force  (SATAF).  On 
programs  not  employing  a  SATAF,  the  evalu¬ 
ators  may  work  with  the  prime  civil  engi¬ 
neering  activity  responsible  for  the  survey 
and  planning  of  the  facilities. 

(2)  The  contractor’s  facilities  program 
plan  and  the  base  and  MAJCOM  facilities 
program  plans  should  be  reviewed  and  com¬ 
pared  with  test  experience.  AFM  86-2  com¬ 


putations  may  be  used  when  applicable. 
Quantitative  inputs  for  these  computations 
should  come  from  the  results  of  the  reliabil¬ 
ity,  maintainability,  and  manpower  evalua¬ 
tion  objectives. 

(3)  Maintenance  activities  should  be 
monitored  and  reviewed  periodically  in  light 
of  facilities  requirements  to  identify  and 
report  any  unique  requirements,  new  facili¬ 
ties,  additions,  or  modifications  needed  to 
support  the  system. 

(4)  A  questionnaire  may  be  used  to  sum¬ 
marize  opinions  in  the  area. 

c.  Data  Requirements.  Data  require¬ 
ments  will  typically  include  the  following: 

(1)  Contractor’s  facilities  program  plan. 

(2)  Base  and  MAJCOM  facilities  program 
plan. 

(3)  Data  elements  required  to  accomplish 
AFM  86-2  computations.  (Obtained  from 
reliability,  maintainability,  and  manpower 
objectives  and  SE  tables  of  allowances.) 

(4)  Minutes  of  site  activation  conference, 
meetings,  and  working  groups. 

(5)  Results  of  facility  evaluation  ques¬ 
tionnaire. 

d.  Assessment.  The  assessment  may 
include  review  of  programmed  facilities 
requirements  in  light  of  test  experience  and 
review  of  activities  to  identify  any  unique, 
new,  or  altered  facilities  requirements  which 
have  not  been  previously  identified  or  pro¬ 
grammed. 

e.  Assessment  Criteria.  Compare  pro¬ 
grammed  facilities  versus  computed  require¬ 
ments,  with  an  assessment  as  to  whether 
adequate  facilities  are  available  or  being 
programmed. 

3-9.  Maintenance  Training.  The  assess¬ 
ment  effort  will  be  to  assess  whether  Air 
Force  maintenance  personnel  with  system 
training  can  maintain  and  support  the  sys¬ 
tem  in  its  intended  environment.  This 
includes  as  assessment  of  specialties  and 
skill  levels  required  to  perform  base-level 
tasks,  as  well  as  the  need  for  new  specialties 
or  unique  training  requirements  for  existing 
specialties  to  support  system-unique  require¬ 
ments.  It  also  includes  assessing  the  mainte¬ 
nance  training  conducted  prior  to  and  in 
support  of  OT&E  and  providing  information 
to  assist  in  refining  training  requirements, 
technical  training  materials,  and  facilities 
required  to  support  systems  during  opera¬ 
tional  use.  Air  Training  Command  (ATC) 
provides  the  training  evaluations. 

a.  General  Considerations.  Training 
assessments  include  the  following: 

(1)  Assess  whether  any  aspects  to  the 
system  will  impose  adverse  or  unreasonable 
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trainin g  support  requirements  beyond  those 
generally  acceptable  to.  and  standard  within, 
the  technical  training  community. 

(a)  System  aspects  include  design  or 
construction  of  subsystems,  operational  sup¬ 
port  equipment,  software,  and  technical  data; 
maintenance  or  logistics  concepts;  and  quanti¬ 
tative  and  qualitative  personnel  require¬ 
ments. 

(b)  Training  support  requirements  in¬ 
clude  training  development  lead  times  or 
procedures,  course  lengths,  technical  training 
material  suitability,  quantities  and  costs, 
facilities,  instructors,  and  logistics  support. 

(2)  Assess  the  ability  of  the  training 
program  being  developed  (including  instruc¬ 
tion  and  course  preparation/contents,  techni¬ 
cal  training  material  identification  and  pro¬ 
curement  actions,  support  requirements  for 
technical  data,  facilities  and  logistics  support, 
and  associated  scheduling  actions)  to  match 
the  using  and  supporting  commands’  require¬ 
ments  for  training  maintenance  personnel. 

b.  Test  Methodology.  The  methodology 
for  the  assessment  should  include  the  follow¬ 
ing  activities  by  the  evaluators: 

(1)  Review  training  plans,  course  docu¬ 
ments,  technical  data,  and  system  test  re¬ 
sults  for  impact  on  training  support. 

(2)  Conduct  interviews,  as  required,  in 
support  of  test  objectives. 

(3)  Conduct  over-the-shoulder  observa¬ 
tions  of  tasks  being  accomplished  during  test 
activities. 

(4)  Participate  in  the  service  reporting 
system. 

(5)  Conduct  interviews  with  supervisory 
and  maintenance  personnel  to  determine  if 
training  provided  in  support  of  OT&E  ade¬ 
quately  prepared  the  graduates  to  accom¬ 
plish  the  T&E  objectives. 

c.  Data  Requirements: 

(1)  Data  requirements  for  the  evaluation 
may  include  the  following: 

(a)  Contractor  maintenance  instructions/ 
preliminary  technical  data. 

(b)  Logistics  support  analysis  reports 
(LSA-002,  -Oil,  and  -014). 

(c)  Maintenance  concepts. 

(d)  Instructional  systems  development 
(ISD)  data  base  and  documents. 

(e)  Training  documents  (plans,  training 
standards,  plans  of  instructions,  course 
charts,  etc.). 

(f)  Air  Force  training  regulations  and 
manuals. 

(g)  Target  population,  training  program 
requirements,  and  other  training  manning 
documents. 

(2)  References: 

(a)  AFR  50-9,  Special  Training. 


ib)  AFR  50-18,  Interservice  Training. 

(c)  AFR  55-43.  Management  of  Opera¬ 
tional  Test  and  Evaluation. 

( d)  ATCR  800-1.  Program  Management, 
and  volumes  I  and  II.  ATC  Participation  in 
Systems  Acquisition. 

(e)  AFLCP/AFSCP  800-34.  Acquisition 
Logistics  Management. 

d.  Assessment  and  Assessment  Crite¬ 
ria.  Quantitative  criteria  are  not  normally 
available  to  address  this  area.  The  assess¬ 
ment  will  be  based  on  the  subjective  opinions 
of  test  team  maintenance  and  training  per¬ 
sonnel  as  to  the  ability  of  the  training  pro¬ 
gram  to  support  the  system. 

3-10.  Maintenance  Planning.  Mainte¬ 
nance  planning  assesses  the  adequacy  of  all 
actions  defined  for  each  significant  main¬ 
tenance  task  required  to  support  the  weapon 
system.  Specifically,  it  is  the  assessment  of 
the  planning  for  all  the  activity  required  to 
achieve,  restore,  or  maintain  the  operational 
capability  of  the  system  or  equipment. 
MOEs  for  this  objective  will  be  the  subjective 
assessments  (backed  up  by  qualitative  and 
quantitative  data  from  the  other  objectives) 
of  the  adequacy  of  maintenance  and  logistics 
planning  to  provide  the  required  support. 

a.  General  Considerations.  The  follow¬ 
ing  factors  should  be  considered  in  developing 
objectives  and  MOEs: 

(1)  The  ability  to  effectively  and  effi¬ 
ciently  support  the  weapon  system. 

(2)  The  suitability  of  repair-level  deci¬ 
sions. 

(3)  The  use  of  organic,  interim  contrac¬ 
tor  support  (ICS)  or  contractor  logistics  sup¬ 
port  (CLS)  resources  for  organizational-, 
intermediate-,  and  depot-level  hardware  and/ 
or  software  support. 

(4)  The  ability  to  adequately  support 
deployment  requirements. 

(5)  The  adequacy  of  integrated  diag¬ 
nostics  concepts. 

(6)  The  validity  of  the  assumptions  the 
maintenance  concept/plan  was  based  on. 

b.  Test  Methodology.  The  methodology 
for  this  objective  may  include  the  following: 

(1)  A  comparison  of  the  logistics  factors 
used  to  compute  the  RLA  and  SMR  codes 
with  conditions  actually  being  experienced  or 
projected  for  the  mature  environment. 

(2)  A  comparison  of  the  key  programmed 
suitability  performance  parameters  with  those 
actually  experienced  or  projected  and  the 
planning  actions  being  taken  to  adjust  or 
accommodate  differences.  Consider  such  fac¬ 
tors  as  the  following: 

(a)  R&M  performance. 

(b)  Integrated  diagnostics  capability. 
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<c)  Supply  provisioning. 

(d)  Technical  data  adequacy. 

(e)  Support  equipment  suitability. 

(3)  Changes  in  the  maintenance  concepts 
and  appropriate  adjustments  or  changes  in 
maintenance  planning. 

(4)  Review  of  all  the  suitability  objectives 
for  problems  and  the  adequacy  of  mainte¬ 
nance  planning  to  overcome  or  compensate 
for  those  problems,  when  applicable. 

c.  Data  Requirements.  Data  for  this 
evaluation  will  typically  come  from  the  other 
suitability  objectives.  If  ICS  is  planned  until 
the  system  reaches  initial  operational  capabil¬ 
ity  (IOC),  the  transition  plan  to  organic 
maintenance  must  be  obtained  and  reviewed. 

d.  Assessment.  The  assessment  should 
be  designed  to  identify  areas  where  mainte¬ 
nance  planning  is  not  adequate  to  support 
the  required  level  of  mission  performance  and 
to  make  appropriate  recommendations.  As 
a  minimum,  these  areas  should  be  assessed: 

(1)  The  ability  of  the  maintenance  plan¬ 
ning  to  result  in  the  necessary  actions  and 
support  to  ensure  the  system  or  equipment 
attains  required  operational  capability. 

(2)  The  specification  and  realism  of  crite¬ 
ria  for  repair  times,  maintainability  and 
reliability  characteristics,  SE  requirements, 
maintenance  skills,  and  facilities  require¬ 
ments. 

(3)  The  adequacy  of  the  RLA  and  wheth¬ 
er  the  most  efficient  and  economical  repair 
levels  have  been  established.  (This  evalua¬ 
tion  can  also  be  supported  by  quantitative 
data  from  other  appropriate  objectives.) 

(4)  The  scope  and  completeness  of  transi¬ 
tion  plans  designed  to  facilitate  transfer  of 
logistics  support  from  contract  to  organic 
capabilities. 

(5)  For  CLS,  provisions  for  adequate 
documentation,  source  code,  and  skills/ 
experience  levels  may  be  assessed  to  identify 
potential  hardware  or  software  problem  areas 
that  could  affect  system  support,  configura¬ 
tion  management,  or  mission  performance. 

e.  Assessment  Criteria.  The  assessment 
criteria  may  be  linked  to  the  criteria  from 
the  availability  objective.  Whenever  possible, 
however,  the  test  team  member  should  use 
quantitative  date  to  support  findings. 

3-11.  Other  Considerations.  Although  the 
following  factors  are  not  normally  categorized 
as  operational  suitability  objectives,  they  may 
play  a  significant  role  in  estimating  a  sys¬ 
tem’s  wartime  capability.  Many  of  them 
apply  to  more  than  one  of  the  preceding 
elements  (supply,  support  equipment,  etc.). 


a.  Interoperability  is  the  ability  of  sys¬ 
tems,  units,  or  forces  to  provide  services  to. 
and  accept  services  from,  other  system,  units, 
or  forces  and  to  use  the  services  so  ex¬ 
changed  to  enable  them  to  operate  effectively 
together  (AFR  80-14).  In  a  logistics  perspec¬ 
tive,  interoperability  may  be  viewed  from  a 
system  or  subsystem  viewpoint. 

b.  Compatibility  is  the  capability  of  two 
or  more  operational  items/systems  to  exist  or 
function  as  elements  of  a  larger  operational 
system  or  operational  environment  without 
mutual  interference  (AFR  80-14).  Compati¬ 
bility  may  also  be  viewed  from  a  system  or 
subsystem  perspective. 

c.  Wartime  usage  rates  are  the  rates  at 
which  system  and  their  supporting  subsys¬ 
tem,  SE,  and  spares  are  consumed/used 
under  war  conditions.  Operational  suitability 
evaluations  should  be  done  in  context  of  the 
planned  wartime  usage  rates. 

d.  The  use  of  chemical,  biological,  radio¬ 
logical  (CBR)  warfare  protective  clothing  and 
equipment  must  be  considered  when  estimat¬ 
ing  the  utility  of  system  operated  under  CBR 
threat  conditions. 

e.  Human  factors  is  a  body  of  scientific 
facts  about  human  characteristics.  The  term 
covers  all  biomedical  and  psychosocial  con¬ 
siderations.  It  includes,  but  is  not  limited  to. 
principles  and  applications  in  the  areas  of 
human  engineering,  personnel  selection, 
training,  life  support,  job  performance  aids, 
and  human  performance  evaluation  (MIL-STD 
721).  Maintainability  evaluations  should 
take  into  account  the  impact  of  human 
factors  on  ease  of  maintenance,  accessibility, 
and  similar  considerations. 

f.  Depot  support  of  emerging  systems  is 
often  immature  during  OT&E.  As  a  result, 
assessments/evaluations  of  this  area  are  not 
performed,  except  for  a  review  of  planning 
documents.  Depot  activities  may  be  modeled 
to  determine  impact  on  field  operations  and 
maintenance.  Such  modeling  should  be 
planned  for,  coordinated,  and  verified  (if  not 
validated)  prior  to  the  start  of  test.  Test 
planners  and  test  teams  should  be  alert  to 
the  need  for  adequate  depot  planning  and 
should  review  available  data. 

g.  Safety  is  freedom  from  conditions  which 
can  cause  death,  injury,  occupational  illness, 
or  damage  to  or  loss  of  equipments  or  prop¬ 
erty.  Formal  safety  assessments  are  nor¬ 
mally  conducted  by  HQ  AFOTEC/SE,  but 
safely  considerations  should  be  included  in 
maintainability  or  logistics  supportability 
assessments. 
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Part  EE 

KEY  OPERATIONAL  SUITABILITY  ACTIVITIES 
Chapter  4 

PREPARING  FOR  TEST  PLANNING  REVIEWS  (TPR) 


4-1.  General: 

a.  Test  planning  reviews  are  major  parts 
of  a  dynamic  and  evolving  AFOTEC  process 
of  developing  a  test  plan.  The  process  begins 
with  developing  a  test  concept  and  a  descrip¬ 
tion,  including  rationale  and  assumptions,  of 
the  test  structure,  evaluation  methodology, 
and  management  approach  that  will  be  used 
to  evaluate  the  operational  effectiveness  and 
suitability  of  a  weapon  system.  It  should  be 
understood  the  test  concept  will  mature  with 
changes  in  the  program  and  will  eventually 
transition  into  the  OT&E  test  plan. 

b.  TPRs  provide  a  structure  and  meth¬ 
odology  for  planning  an  executable  OT&E. 
They  ensure  continuing,  consistent,  and 
compressive  staff  review  of  the  test  concept 
and  adequacy  of  test  planning.  In  addition, 
they  provide  feedback,  direction,  and  assis¬ 
tance  to  the  test  support  groups  (TSG). 

c.  There  are  currently  three  TPRs  (each 
may  be  preceded  by  an  internal  LG  review 
with  material  presented  limited  to  suitability 
test  and  evaluation).  The  AFOTEC  Com¬ 
mander  is  the  approval  authority  for  all  test 
planning  conducted  during  the  TPR  process. 

(1)  TPR  #1:  Test  Concept  Approval. 

(2)  TPR  #2:  Test  Concept  Update. 

(3)  TPR  #3:  Draft  Test  Plan. 

d.  Figure  4-1  depicts  the  role  of  TPRs  in 
the  OT&E  planning  process. 

4-2.  Test  Concept  Approval: 

a.  From  the  time  a  work  directive  is 
issued  and  the  test  manager  convenes  a  TSG, 
the  suitability  test  planner  will  become 
deeply  involved  in  thinking,  reading,  discuss¬ 
ing,  and  learning  about  the  system  needed, 
the  system  expected  for  OT&E,  and  organi¬ 
zational  and  procedural  details  of  OT&E 
itself.  The  efforts/tasks  before  the  planner 
may  seem  monumental,  especially  if  the 
planner  is  new  to  the  OT&E  community. 
Although  training  in  OT&E  is  available,  the 
suitability  test  planner  may  find  it  too  gener¬ 
al  to  assist  in  tackling  each  specific  task. 
Therefore,  interface  with  one’s  peers,  super¬ 
visors,  and  the  TPR  committee  members  is 
a  must  for  successful  development  of  a  test 
concept. 

b.  In  preparing  for  TPR  #1,  the  TSG  must 
develop  a  feasible  test  concept  that  includes 


user  requirements,  operational  environment, 
focus  of  test,  test  scenarios,  and  test  re¬ 
sources.  These  five  areas  may  be  reviewed 
independently  or  concurrently  by  involved 
AFOTEC  division  chiefs,  AFOTEC/XP,  and 
AFOTEC/CN  in  turn. 

(1)  Requirements  Review.  Operational 
requirements,  derived  from  key  program 
documents  <e.g.,  MNS,  ORD,  and  COEA),  are 
evaluated  on  three  aspects:  completeness, 
relevance,  and  operational  testability.  As¬ 
sumptions  necessary  to  bridge  voids  resulting 
from  insufficient  information  and  program 
immaturity  must  be  made  and  tracked  until 
resolved.  General  considerations  for  this 
review  include  the  following: 

(a)  Are  requirements  expressed  in 
operationally  relevant  terms  (reference  AFP 
52-9)? 

(b)  Are  requirements  clearly  defined, 
either  qualitatively  or  quantitatively,  to 
enable  development  of  comprehensive  opera¬ 
tional  test  criteria? 

(c)  Are  requirements  complete  enough 
to  accomplish  the  stated  operational  mis¬ 
sions?  Is  each  ILS  element  addressed? 

(d)  Do  the  requirements  account  for  the 
operational  environment  in  which  the  system 
will  be  deployed? 

(e)  Are  the  key  (quantitative)  require¬ 
ments  reflected  in  the  requirements  correla¬ 
tion  matrix  (RCM)? 

(f)  Are  there  unique  requirements  (soft¬ 
ware,  diagnostics,  etc.),  and  if  so,  are  they 
clearly  defined? 

(g)  Is  each  requirement  testable  during 
OT&E,  and  if  not,  is  there  a  workaround? 

(2)  Operational  Environment  Review. 
The  operational  environment  review  should 
ensure  the  TSG  understands  the  operational 
missions  and  the  entire  range  of  operating 
conditions  when  focusing  the  test  effort.  The 
review  is  a  survey  of  the  concept  of  opera¬ 
tions,  maintenance  concept  (or  logistics  con¬ 
cept),  and  threat  environment.  General 
considerations  for  this  review  include: 

(a)  Do  the  operational  mission  scenarios 
pose  unique  challenges  for  logistics  support 
during  OT&E  or  during  deployment? 

(b)  Does  the  maintenance  concept  re¬ 
flect  current  Air  Force  policy,  and  can  it  be 
represented  during  OT&E?  If  not,  is  contrac- 


CM  POLICY  LTR 


Figure  4-1.  OT&E  Planning  Process 


AFOTECP  400-1  15  May  1991 


4-3 


tor  logistics  support  (CLS)  the  only  alterna¬ 
tive  or  are  there  workarounds  to  minimize/ 
avoid  CLS?  What  is  the  impact  of  not  using 
CLS? 

Note:  Title  10,  US  code.  Section  2399  re¬ 
stricts  system  contractor  involvement  in 
OT&E.  In  order  to  comply  with  the  intent 
if  not  the  letter  of  the  law,  suitability  test 
planners  should  avoid  CLS  during  active  field 
testing,  unless  such  support  is  identified  for 
the  deployed  system,  or  it  is  not  feasible  to 
use  government  (organic)  support  resources. 

(c)  Are  the  skill  levels  and  number  of 
maintenance  personnel  required  for  the 
system’s  life  specified?  What  minimum  skills 
and  maintenance  manpower  are  required  for 
test? 

(d)  What  are  the  deployed  maintenance 
conditions?  Which  conditions  are  planned  for 
the  test? 

ie)  Are  maintainability  demonstrations 
required?  Can  start  and  stop  times  for 
maintenance  be  defined? 

(f)  What  role  does  integrated  diag¬ 
nostics  have  within  the  system’s  maintenance 
concept  and  concept  of  operations? 

(g)  What  type  of  technical  data  is 
specified:  paper,  digital,  or  both?  Which 
will  be  available  for  OT&E? 

(h)  What  are  the  main  suitability 
drivers  for  the  system? 

(3)  Focus  of  Test  Review.  It  is  recog¬ 
nized  that  OT&E  may  not  be  able  to  test  all 
aspects  of  the  system;  however,  the  suitabil¬ 
ity  test  planner  should  extract  those  elements 
critical  to  decision  makers  and  evaluate  the 
system  based  on  its  capability  to  accomplish 
those  critical  elements  under  operational 
conditions.  This  review  consists  of  identify¬ 
ing  the  mission  critical  elements  (MCE) 
related  to  mission  conditions,  the  critical 
operational  issues  (COI)  derived  from  the 
MCEs  and  measures  that  will  provide  the 
data  to  support  an  evaluation  of  the  critical 
operational  issues  (reference  figure  4-2). 
COIs  are  developed  from  user-identified 
critical  system  effectiveness  and  suitability 
requirements  and  are  focused  or  established 
at  the  mission  level.  If,  for  example,  reliabil¬ 
ity  is  an  MCE  for  Mission  X,  there  should 
not  be  a  COI  for  "the  reliability  of  the  sys¬ 
tem  during  Mission  X"  (this  should  be  an 
objective).  Rather,  the  COI  would  be  worded 
to  ask  how  well  the  system  performs  (or  how 
capable  the  system  is)  during  Mission  X  with 
consideration  given  to  reliability  and  any 
other  element  of  suitability  or  effectiveness. 
Hence,  suitability-related  test  objectives  may 
apply  to  all  COIs,  instead  of  specific  ones. 
Activities  for  this  review  should  include: 

(a)  Contact  the  system’s  user  to  discuss 


MCEs  and  identify  COIs. 

(b)  Work  with  the  TSG  to  limit  the 
COIs  to  those  clearly  relevant  to  the  system's 
intended  missions. 

(c)  Begin  thinking  about  how  the  COIs 
will  be  answered,  what  objectives  are  neces¬ 
sary,  what  measures  will  provide  data  to 
support  an  evaluation  of  the  COIs,  what 
long-lead  resources  may  be  needed,  etc. 

(d)  Begin  programming  test  resources 
with  the  Resource  Development  Division 
(RMX). 

(4)  Test  Scenario  Review.  Test  scenar¬ 
ios  should  replicate  MCEs  and  conditions  of 
the  operational  environment  (mission  scenar¬ 
io)  to  the  maximum  extent  feasible.  This 
review  includes  all  planned  OT&E  test  activ¬ 
ities  and  must  provide  traceability  from  the 
focus  of  test  to  the  proposed  test  scenarios. 
The  review  must  also  provide  the  TSG’s 
evaluation  methodology  approach,  including 
the  planned  confidence  level  and  resulting 
scope  of  test.  Considerations  for  this  review 
include: 

(a)  How  well  will  the  planned  test 
environment  replicate  the  intended  operation¬ 
al  environment? 

(b)  Are  there  limitations  caused  by  lack 
of  resources,  schedule  constraints,  test  article 
availability,  configuration,  etc.?  If  so,  what 
are  the  impacts?  Are  there  workarounds  to 
reduce  the  impact? 

(c)  If  an  area  cannot  be  field  tested,  are 
there  alternate  methods  for  evaluation  (e.g., 
simulators,  models,  studies,  etc.)? 

(5)  Test  Resource  Review.  Test  resources 
are  the  assets  AFOTEC  needs  available  to 
complete  OT&E.  These  assets  include  test 
articles,  ranges,  threat  simulators,  test  team 
makeup,  and  anything  else  required  to  carry 
out  a  thorough  OT&E.  During  this  review, 
the  TSG  must  identify  test  capability  short¬ 
falls  and  changes  to  test  scenarios  because  of 
lack  of  existing/programmed  test  resources. 
Considerations  for  this  review  include: 

(a)  Is  there  a  test  program  outline 
(TPO)? 

(b)  Can  assets  be  shared  among  other 
tests  at  the  test  site  or  with  other  agencies? 

(c)  Is  the  data  collection  system  for 
suitability  OT&E  defined  and  will  it  be 
available  at  the  test  site  before  the  start  of 
test? 

(d)  Are  the  quantities  of  test  articles, 
people,  supplies,  etc.,  reasonable?  Can  you, 
the  suitability  test  planner,  justify  these 
quantities? 

(e)  Are  the  appropriate  data  item 
descriptions,  statement  of  work  tasks,  and 
system  specifications  included  in  the  system 
contract  to  allow  delivery  of  software  data 
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f  documentation,  source  code,  software  defi¬ 
ciency  reports;  for  AFOTECP  800-2  series 
evaluations? 

c.  Assumptions  made  during  test  concept 
development  to  bridge  voids  resulting  from 
insufficient  information  and  program  im¬ 
maturity  must  be  highlighted  at  TPR  #1. 
As  the  test  planning  process  progresses,  the 
test  planner  will  refine  the  test  concepts  to 
reflect  a  maturing  program.  Also  included 
are  proposed  dates  for  completing  significant 
planning  activities  (e.g.,  operational  assess¬ 
ments). 

d.  The  focus  at  TPR  #1  (as  well  as  TPR 
#2  and  TPR  #3)  will  be  on  content,  not  on 
the  format  of  the  presentation. 

4-3.  Updating  the  Test  Concept: 

a.  The  primary  purpose  of  TPR  #2  is  to 
finalize  the  test  concept.  Programs  will 
normally  transition  from  XP  to  TE  at  comple¬ 
tion  of  TPR  #2.  The  TSG  will,  after  TPR  #1, 
continue  to  define,  refine,  and  validate  the 
test  concept,  resolving  the  issues  and  as¬ 
sumptions  previously  identified. 

b.  In  preparing  for  TPR  #2,  the  TSG 
should  review  all  limitations,  constraints, 
issues  or  concerns,  and  action  items  being 
taken  to  resolve  them.  The  TSG  should  be 
able  to: 

(1)  Discuss  how  and  when  the  staff  may 
become  involved  in  the  resolution  process. 

(2)  Identify  key  activities/information 
which  have  become  available  since  the  previ¬ 
ous  TPR. 

(3)  Discuss  the  status  of  assumptions. 

(4)  Present  a  risk  assessment  of  getting 
"undetermined"  test  results  for  user  suitabil¬ 
ity  requirements. 

(5)  Identify  missing  contractual  require¬ 
ments  that  will  prevent  the  conduct  of 
AFOTECP  800-2  series  evaluations.  Describe 
what  approach  will  be  taken  to  resolve  the 
software  evaluation  issue. 

c.  After  TPR  #2,  the  TSG  will  prepare  a 
scripted  executive-level  test  concept  briefing 
and  formally  present  it  to  the  directors. 
After  presentation  to  the  directors,  the  brief¬ 
ing  is  presented  to  CN  and  CV/CC,  in  turn. 

4-4.  Draft  Test  Plan  (with  DMAP): 

a.  The  primary  purpose  of  TPR  #3  is  to 
allow  the  TPR  committee  to  review  and 
advise  the  TSG  in  development  of  the  draft 
OT&E  plan. 

b.  Before  TPR  #3  the  draft  test  plan, 
DMAP,  ORD,  TEMP,  etc.,  are  made  available 
to  the  logistics  TPR  committee  member. 

c.  In  preparing  for  TPR  #3,  the  TSG 
should  follow  the  guidance  necessary  to 
develop  a  draft  OT&E  plan  and  appropriate 


supplements  found  in  AFR  55-43.  The 
DMAP,  a  separate  document  from  the  test 
plan,  should  be  developed  to  the  point  of  sup¬ 
porting  the  draft  test  plan  (see  chapter  7). 
The  TSG  should  be  able  to: 

(1)  Explain  the  status  of  OT&E  planning 
and  provide  rationale  for  the  detail  which 
has  been  added  since  the  last  TPR. 

(2)  Explain  any  departures  between  the 
proposed  draft  test  plan  and  the  approved 
test  concept. 

d.  After  TPR  #3,  the  TSG  will  again  brief 
the  directors,  CN,  and  CV/CC  in  turn. 

4-5.  TPR  Example: 

a.  An  example  of  a  test  concept  is  being 
developed  in  the  AFOTEC  Test  Concept 
Development  Handbook.  In  the  interim,  the 
best  source  for  information  concerning  con¬ 
tent  of  a  test  concept  is  from  recently  com¬ 
pleted  TPRs.  In  addition,  LG  reference 
library  contains  extracts  from  past  TPRs; 
however,  because  the  OT&E  planning  process 
is  dynamic,  the  suitability  test  planner 
should  be  thoroughly  familiar  with  the  latest 
policy  and  procedures/guidance  on  TPRs  and 
any  specific  direction  from  the  test  manager. 

b.  An  example  of  a  draft  test  plan  is  con¬ 
tained  in  AFR  55-43.  Examples  of  test  plan 
supplements  including  data  management  and 
analysis  plans  are  available  in  the  AFOTEC 
archives. 

4-6.  Lessons  Learned: 

a.  Test  Concept  Development: 

(1)  Know  the  program  thoroughly  (key 
decision  milestones,  technical  features,  acqui¬ 
sition  strategy,  etc.).  Plan  test  events  to 
provide  reports  for  key  decision  milestones. 

(2)  A  good  suitability  approach  is  a 
cooperative  effort.  Get  everybody  involved. 
Pass  early  drafts  around  your  branch  for 
comment.  Check  with  personnel  who  have 
recently  formulated  briefings  for  the  most 
recent  guidance  and  suggestions.  The  earlier 
you  get  feedback,  the  better. 

(3)  Have  the  test  manager  involved,  even 
in  internal  LG  suitability  reviews.  He/she 
can  also  answer  overall  program  and  opera¬ 
tional  effectiveness  questions  (highly  desir¬ 
able  for  division-level  briefings;  essential  for 
directorate  level). 

(4)  For  the  reliability  and  availability 
objectives,  differentiate  between  the  state¬ 
ment  of  the  objective  and  the  MOE.  For 
example,  weapon  system  reliability  (WSR) 
should  be  an  MOE,  not  an  objective.  The 
term  WSR  should  not  be  used  as  an  objective 
(as  in  "Evaluate  High  Speed  Antiradiation 
Missile  (HARM)  WSR").  When  used  as  an 
MOE,  WSR  should  be  explicitly  defined  (i.e.. 
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at  what  point  in  a  mission  does  the  measure 
start  and  when  does  it  stop).  Likewise,  for 
the  availability  objective,  "availability''  is  not 
an  MOE. 

(5)  Examine  carefully  the  issue  of  dor¬ 
mancy  and  its  impact  on  reliability.  Use  a 
separate  objective  for  "dormant  reliability" 
only  when  dormant  storage  is  a  significant 
factor  in  the  operational  use  of  the  item  (i.e., 
a  "wooden-round"  munition).  Some  items 
(such  as  the  HARM)  have  long  periods  of 
storage  between  operational  uses  but  receive 
a  checkout  before  being  used.  If  you  believe 
it  worthwhile  to  examine  the  impact  of  these 
storage  periods  on  reliability,  include  this 
examination  in  the  methodology  of  the  relia¬ 
bility  objective.  See  attachment  2  for  discus¬ 
sion  on  dormant  reliability. 

(6)  Look  carefully  at  objectives  for  which 
you  have  selected  numerous  MOEs.  In  some 
cases  the  MOEs  can  be  generalized  to  a 
single  MOE  and  the  various  aspects  of  the 
MOE  incorporated  in  the  evaluation  method. 
For  example,  the  two  MOEs  "mean  loading 
and  checkout  time  for  a  HARM"  and  "mean 
loading  and  checkout  time  for  gravity  weap¬ 
ons"  can  be  combined  into  a  single  MOE 
"mean  loading  and  checkout  time"  with  the 
variables  included  in  the  method. 

(7)  Get  your  requirements  for  OT&E  (e.g., 
Air  Force  hands-on  maintenance,  contractor 
support)  into  the  request  for  proposal  (RFP). 
Talk  with  SPO  personnel  so  they  understand 
and  support  the  OT&E  requirements.  Follow 
through  to  make  sure  your  requirements  are 
included  in  the  contract.  Provide  whatever 
support  you  can  to  make  sure  the  require¬ 
ments  do  not  get  negotiated  out  before  the 
final  contract  is  signed.  This  is  critical  for 
requiring  the  contractor  to  use  data  forms/ 
equipment  peculiar  to  government  data 
collection  systems  and  requiring  the  contrac¬ 
tor  to  use  the  Service  Reporting  System  (TO 
00-35D-54).  See  LGOI  400-6,  Request  for 
Proposal  (RFP)  Checklist,  for  further  ideas. 

(8)  Be  thorough  in  responding  to  SPO 
data  calls  to  ensure  you  request  the  right 
data  from  the  contractor.  Be  concerned  with 
only  the  data  you  need.  Review  logistics 
support  analysis  deliverables  and  select 
reports  that  will  be  needed  to  support  your 
test  objectives. 

(9)  Ensure  enough  assets  are  procured 
for  test  and  the  delivery  schedules  for  sup¬ 
port  equipment,  training,  technical  data,  etc., 
will  support  a  thorough  suitability  evaluation. 
Do  not  automatically  accept  the  number  of 
items  the  SPO  says  will  be  available  for  test. 
Determine  what  you  need  and  work  to  get 
that  number  of  items.  If  you  have  to  accept 
fewer  items,  provide  as  part  of  the  approach 


(under  limiting  factors)  how  this  compromise 
will  impact  the  validity  of  the  test  results. 

(10)  Study  and  influence  the  test  condi¬ 
tions  under  which  contractual  R&M  demon¬ 
strations  are  conducted  so  that  they  produce 
operationally  relevant  data. 

(11)  Agreement  on  system  design,  con¬ 
figuration,  and  testing  concept  is  difficult  to 
achieve  among  the  R&D  organization,  test 
agency,  using  command,  and  supporting  com¬ 
mand  during  the  concept  exploration  phase. 
The  logistics  evaluation  manager  must  plan 
for  this  condition. 

(12)  Data  availability  for  evaluation  is 
limited.  Agreement  as  to  what  data  will  be 
available  and  from  whom  is  difficult  to 
achieve.  Because  of  this,  evaluations  will 
initially  be  limited  to  best  judgment  or 
expert  opinions. 

b.  Measures  of  Effectiveness  to  Satis¬ 
fy  Test  Objectives: 

(1)  Test  planning  depends  on  developing 
MOEs  that  are  directly  applicable  to  the  test 
objectives.  The  test  planner  must  select 
MOEs  which  directly  relate  to  the  system  s 
ability  to  accomplish  its  mission.  Selected 
MOEs  should  also  be  sensitive  to  mission 
impacts  caused  by  particular  subsystems  or 
procedures. 

(2)  When  establishing  MOEs,  recognize 
which  events  occur  during  peacetime,  war¬ 
time,  and  the  transition  from  peacetime  to 
wartime.  For  many  systems,  the  employ¬ 
ment,  deployment,  and  logistics  support 
concepts  are  implemented  in  a  fully  pre¬ 
planned  and  controlled  sequence  of  events. 
This  sequence  starts  at  a  random  time  dur¬ 
ing  normal  peacetime  operations,  passes 
through  an  increased  readiness  stage,  and 
terminates  with  either  an  execution  order  or 
an  order  to  cancel  the  increased  preparedness 
conditions.  While  it  may  be  appropriate  to 
use  average  times  for  MOEs  (e  g.,  MTBM)  to 
describe  a  system’s  capability  during  peace¬ 
time  operations,  it  may  or  may  not  be  appro¬ 
priate  to  use  averages  to  describe  the  sys¬ 
tem’s  capability  in  wartime  or  surge  situa¬ 
tions.  In  this  case,  it  may  be  necessary  to 
define  two  levels  of  MOEs. 

(3)  The  test  planner  must  always  keep 
in  mind  the  feasibility  of  potential  MOEs. 
Data  management  and  analysis  requirements 
are  driven  by  MOE  selections.  The  data 
management  and  analysis  plan  (DMAP) 
(discussed  in  chapter  7)  describes  the  test 
data  to  be  collected,  how  they  will  be  col¬ 
lected,  and  how  they  will  be  processed  to 
support  each  MOE.  MOEs  must  be  calcu¬ 
lated  on  the  basis  of  obtainable  test  data. 
This  means  the  test  planner  should  keep  in 
mind  whether  the  necessary  data  and  calcu- 
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lation  and  analysis  techniques  will  be  avail¬ 
able  at  the  proper  time  during  test. 

(4)  Operational  suitability  evaluations 
encompass  a  hierarchy  of  critical  OT&E 
issues,  objectives,  and  MOEs  which  are 
interrelated  vertically  and  horizontally.  The 
planner  should  establish  evaluation  criteria 
at  the  system  level  for  the  fewest  number  of 
MOEs  that  can  satisfy  the  decision  maker's 
concerns.  Circumstances  may  arise,  however, 
when  it  is  desirable  to  establish  MOEs  and 
evaluation  criteria  which  focus  management’s 
attention  on  a  particular  area  of  concern, 
c.  Logistics  Evaluation  Management: 

(1)  Logistics  evaluation  managers  for 
some  programs  have  had  significant  difficulty 
in  obtaining  user  requirements  that  would 
facilitate  timely  identification  of  COIs  and 
development  of  test  objectives  and  measures 
of  effectiveness. 

(2)  External  influences  and  decisions  can 
have  major  impacts  on  test  planning.  These 
can  range  from  program  starts  and  stops, 
major  policy  shifts,  or  SPO-generated  contract 
changes.  Each  can  radically  change  the 
COIs  and,  therefore,  test  planning.  To  cope 
with  these  pitfalls,  the  logistics  evaluation 
manager  must  stay  abreast  of  programmatic 
developments  and  keep  the  information 
flowing. 

(3)  The  logistics  evaluation  manager  must 
also  cross-check  the  information  acquired  for 
test  planning  to  ensure  it  is  up  to  date  and 
consistent.  Incomplete  operational  require¬ 
ments  data  may  lead  to  inconsistent  contract 
specifications.  Incompatibilities  between 
contract  requirements  and  user  requirements 
are  not  unusual.  In  such  cases,  LG  product 
division  action  is  required  to  elevate  and 
resolve  disconnects,  and  logistics  evaluation 
managers  should  expect  to  see  the  contract 
or  the  user  requirements  changed.  Unfor¬ 
tunately,  it  is  usually  the  user  who  changes, 
since  cost  and  schedule  considerations  usually 
keep  the  SPO  from  changing  the  contract 
unless  the  mission  requirement  in  question 
is  a  paramount  concern  to  the  user.  There¬ 
fore,  the  test  planning  in  this  area  should  be 
particularly  thorough  to  ensure  it  will  with¬ 
stand  the  possible  subsequent  challenge. 

(4)  AFOTEC/LG  action  officers  must 
accurately  forecast  test  requirements  (i.e., 
munitions,  aircraft  spare  parts,  manpower, 
test  equipment,  tooling,  facilities,  vehicles, 
etc.)  as  soon  as  possible  in  the  IOT&E  test 
planning  process  via  the  TPO.  Munitions 
requirements  should  be  forecast  at  least  2 
years  in  advance  so  AFOTEC  requirements 
can  get  into  the  USAF  Tactical  Air  Muni¬ 
tions  Plans  and  Allocation  cycles.  All  other 
logistics  test  resources  must  be  followed  up 


by  the  action  officer  via  message,  letter,  etc., 
between  TPO  cycles  to  ensure  these  OT&E 
critical  test  assets  are  there  when  you  need 
them. 

1 5)  Some  SPOs  have  not  required  contrac¬ 
tors  to  use  SEDS  for  R&M  data  collection 
and  processing  on  the  basis  that  it  costs  too 
much.  Attempt  to  convince  the  SPO  early 
enough  that  SEDS  should  be  used  so  it  is 
not  an  add-on  cost.  Use  AFR  80-14  and 
AFSC  supplement  1  to  AFR  80-14  as  refer¬ 
ences.  If  unsuccessful,  plan  for  the  time  and 
personnel  required  to  translate  the  data  into 
an  Air  Force  (or  government;  data  base. 
This  will  serve  to  limit  reliance  upon  contrac¬ 
tor  data  bases. 

(6)  SPO  acquisition  strategy  can  have 
adverse  impacts  on  test  planning.  Some 
SPOs  have  been  reluctant  to  conduct  mainte¬ 
nance  demonstrations,  establish  joint  reliabil¬ 
ity  and  maintainability  evaluation  teams 
(JRMET),  and  conduct  adequate  integrated 
logistics  support  planning.  The  effect  on  test 
planning  can  be  that  logistics  support  infor¬ 
mation  will  be  lacking  and  OT&E  may  be 
required  to  generate  some  information  which 
would  normally  be  available  from  DT&E. 
The  logistics  evaluation  manager  will  need  to 
expend  extra  effort  to  ensure  the  test  plan 
adequately  addresses  these  possible  deficien¬ 
cies. 

(7)  All  available  skills  within  HQ 
AFOTEC/LG  should  be  brought  to  bear  on 
the  test  planning  process.  The  logistics 
evaluation  manager  should  contact  all  appro¬ 
priate  personnel  within  the  LG  staff  to  draw 
upon  their  experience  in  preparing  the  test 
plan.  One  way  to  do  this  is  to  use  a  team 
approach  within  HQ  AFOTEC/LG.  The  logis¬ 
tics  evaluation  manager,  software  evaluation 
manager,  and  logistics  analyst  should  attend 
the  AFOTEC  TSG  to  increase  their  knowl¬ 
edge  of  the  program  and  reduce  levels  of 
coordination  on  test  planning  documentation. 

(8)  Programs  for  which  test  concepts  (or 
approaches)  were  developed  several  years  in 
the  past  must  be  carefully  scrutinized.  Many 
factors  pertaining  to  the  program  may  have 
changed,  e.g.,  political  interest,  using  com¬ 
mand  need,  and  technology  available  to 
system  concept.  These  factors  could  result  in 
changes  to  COIs.  In  turn,  test  objectives 
may  need  to  be  updated.  Test  objectives 
should  be  redeveloped  if  a  sound  evaluation 
is  doubtful.  Evaluation  should  be  changed 
from  quantitative  to  qualitative  if  available 
data  are  sketchy: 

(9)  The  test  planning  during  the  concept 
exploration  phase  should  be  designed  to 
capture  the  major  system  suitability  drivers. 
For  example,  manpower  limitations  may 
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necessitate  development  of  new  technology  to 
conduct  repairs. 

( 10)  Stability  of  personnel  assigned  to  a 
test  program  is  a  significant  aid  to  devel¬ 
oping  a  sound  test  plan.  Replacements, 
when  necessary,  should  be  well  briefed  before 
assuming  test  planning  responsibilities.  In 
addition,  good  records  of  all  activities  per¬ 
taining  to  the  test  program  should  be  kept 
to  assist  in  continuity  of  the  program. 

(11)  Do  not  let  pride  of  authorship  inter¬ 
fere  with  development  of  ideas.  However,  be 
prepared  to  fully  defend  your  ideas  during 
development  of  the  test  concept.  Questioners 
may  not,  and  probably  do  not,  understand 


the  program  as  thoroughly  as  you  do. 

( 12)  An  AFOTEC  assignment  is.  in  some 
ways,  equivalent  to  a  "mim-Air  Staff’  assign¬ 
ment.  Therefore,  logistics  evaluation  man¬ 
agers  must  have  a  wide  range  of  knowledge. 
They  must  understand  the  organization  and 
functions  of  Office  of  the  Secretary  of  De¬ 
fense  (OSD),  HQ  USAF,  AFSC,  AFLC,  and 
the  operating  commands.  They  must  become 
familiar  with  the  planning,  programming, 
and  budgeting  system  to  ensure  their  re¬ 
source  requirements  are  properly  provided 
for.  They  must  be  constantly  aware  of  the 
roles  and  impacts  these  agencies  and  systems 
can  have  on  test  planning 
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Chapter  5 

PREPARING  FOR  OPERATIONAL  ASSESSMENTS  (OA) 


5-1.  General: 

a.  Operational  Assessments  are  independ¬ 
ent  appraisals  of  the  operational  effective¬ 
ness/suitability  aspects  of  a  system.  Early 
operational  assessments  (EOA)  are  operation¬ 
al  assessments  which  support  major  mile¬ 
stones  and  decision  points  before  Milestone 
II.  Normally,  TSGs  will  conduct  OAs  (usu¬ 
ally  before  Milestone  IIIA)  on  major  defense 
acquisition  programs  with  DOT&E  oversight. 
The  central  theme  is  to  interact  with  the 
operating  command  and  developer  to  ensure 
the  establishment  of  clearly  defined  opera¬ 
tional  requirements  and  meaningful  OT&E 
criteria. 

b.  AFR  55-43  outlines  the  planning  and 
reporting  policy  for  OAs.  Normally  a  plan 
is  not  required;  however,  the  TSG  and  espe¬ 
cially  the  logistics  staff  should  formulate  an 
OA  plan  (even  a  strawman  plan)  to  focus 
the  TSG’s  efforts  and  meet  schedule  mile¬ 
stones  for  reporting  purposes.  OA  reports 
(i.e.,  scripted  briefings)  must  address  six 
areas/objectives: 

(1)  Planning  issues  concerning  readiness 
for  OT&E,  schedule  adequacy,  and  resource 
availability. 

(2)  Status  of  documentation. 

(3)  System  development  and  maturity 
aspects  that  may  impact  the  ability  to  start/ 
complete  OT&E. 

(4)  Identification  of  programmatic  voids. 

(5)  Conclusions  from  special  field  test 
activities. 

(6)  Identification  of  significant  trends. 

c.  The  following  paragraphs  outline  gener¬ 
al  suitability-related  consideration  for  OAs. 
These  considerations  are  based  on  activities 
which  should  be  addressed/complete  before 
the  system  progresses  in  the  defense  acquisi¬ 
tion  process.  The  logistics  staff  should  tailor 
the  considerations  for  their  specific  program’s 
milestone/decision  point.  Assistance/support 
from  implementing,  supporting,  and  using 
agencies  is  permitted  by  AFR  55-43;  however, 
the  rating  should  reflect  an  independent 
AFOTEC  assessment  of  the  system/program. 
There  are  four  rating  categories  as  follows: 

(1)  Green:  No  issue  or  issue/area/objec¬ 
tive  being  adequately  addressed. 

(2)  Yellow:  Issue  requires  additional 
attention. 

(3)  Red:  Significant  areas  of  concern  or 
voids  exist  that  will  impact  OT&E  and/or 
may  prevent  the  achievement  of  specified 
operational  requirements. 


(4)  White:  Issue/area/objective  could  not 
be  assessed. 

5-2.  Milestone  I  OAs.  Historically,  OAs 
for  Milestone  I  have  not  been  required.  It  is 
not  certain  this  trend  will  continue  as  new 
acquisition  policy  and  documents  a*-e  devel¬ 
oped  (e.g.,  cost  and  operational  effectiveness 
analysis).  Although  suitability  considerations 
are  presented  herein  for  possible  Milestone 
I  OAs,  they  apply  equally  well  to  OAs  for 
Milestone  II. 

a.  Planning  Issues.  There  are  no  con¬ 
siderations  for  planning  issues  concerning 
readiness  for  OT&E,  schedule  adequacy,  and 
resource  availability.  This  area  should  be 
assessed  "white." 

b.  Status  of  Documentation.  Con¬ 
siderations  for  this  area  include: 

(1)  Has  the  ORD  been  validated?  Are 
the  R&M  and  operational  requirements 
clearly  stated  and  feasible? 

(2)  Has  the  baseline  logistics  support 
concept  been  validated,  and  have  the  major 
support  items  (hardware  and  software)  been 
identified? 

(3)  Has  the  approved  depot  support 
concept  (hardware  and  software),  including 
maintenance  and  material  support  require¬ 
ments,  been  provided  as  an  annex  to  the 
ORD? 

(4)  Is  the  integrated  logistics  support 
plan  in  place  with  plans  to  conduct  tradeoffs 
among  design  characteristics,  manpower  skill 
levels,  and  support  concepts? 

(5)  Is  the  analysis  of  support  cost  drivers 
complete  (with  target  improvements)  and 
reflected  in  support  resource  requirements? 

(6)  Are  the  operational  and  support 
concepts  complete  with  linkage  to  support 
resource  requirements? 

(7)  Have  R&M  design  parameters  been 
developed  and  compared  to  current  system? 
Has  the  need  for  R&M  growth  management 
plans  been  identified? 

(8)  Have  plans  been  initiated  to  consider 
system  test  and  evaluation,  preplanned 
product  improvements,  program  management, 
etc.? 

(9)  Have  plans  been  initiated  to  ensure 
LCC  disciplines  are  applied  through  the 
acquisition  process? 

(10)  Have  all  software  requirements  for 
OT&E  been  incorporated  in  system  con¬ 
tracts?  Has  the  software  maintenance  con¬ 
cept  been  developed,  and  have  the  associated 
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contractual  requirements  been  placed  on 
contract? 

c.  System  Development  and  Maturity. 
Considerations  for  this  area  include: 

(1)  Are  alternate  concept  studies  com¬ 
plete? 

(2)  Has  a  baseline  system  been  iden¬ 
tified? 

(3)  Have  technology  or  system  concept 
demonstration  been  completed? 

d.  Programmatic  Voids.  Considerations 
for  this  area  include: 

(1)  Has  the  DPML/ILSM,  ALC/SPM  been 
identified? 

(2)  Are  there  sufficient  experienced  logis¬ 
ticians  in  place  with  future  needs  identified 
and  programmed? 

(3)  Has  the  logistics  budget  been  estab¬ 
lished?  Were  standard  factors  used?  Are 
the  logistics  cost  drivers  identified? 

(4)  Do  the  cost  requirements  reflect  the 
overall  baseline  support  concepts? 

(5)  Do  the  logistics  business  strategy 
options  support  the  design  and  support 
options  under  review? 

(6)  Have  ILS  elements  been  given  ap¬ 
propriate  weight  in  source  selection? 

(7)  Is  LSA  on  contract  including  inte¬ 
grated  provisioning  requirements  and  explicit 
tailoring  instructions  to  ensure  LSA  impacts 
the  design  decision  process? 

(8)  Have  product  performance  agree¬ 
ments  or  system  warranties  (see  AFR  70-11) 
been  considered? 

(9)  Have  the  software  maintenance  re¬ 
sponsibilities  been  defined? 

(10)  Have  the  details  of  the  software 
maintenance  concept  been  translated  into  an 
acquisition  contract  line  item  to  procure  the 
software  maintenance/support  capability? 

e.  Special  Field  Test  Activities.  There 
are  no  considerations  for  this  area.  This 
area  should  be  assessed  "white." 

f.  Significant  Trends.  There  are  no 
considerations  for  this  area.  This  area 
should  be  assessed  "white." 

5-3.  Milestone  11  OAs.  This  milestone  is 
usually  the  initial  focus  of  the  TSG’s  opera¬ 
tional  assessment  effort.  Depending  on  the 
program,  the  previous  considerations  may 
apply,  and  some  considerations  presented 
below  may  best  be  deferred  to  Milestone  IIIA. 
Also,  the  considerations  below  may  apply  to 
more  than  one  area. 

a.  Planning  Issues.  Considerations  for 
this  area  include: 

(1)  Have  facility  requirements  been 
identified  and  linked  to  the  budget  to  ensure 
availability  at  need  date? 

(2)  Is  a  provisioning  strategy  in  place 


and  compatible  with  contractor  support 
strategy?  Are  review  schedules  developed? 

(3)  Has  the  need  for  a  postproduction 
support  plan  been  identified? 

(4)  Do  ILS  elements  continue  to  be  given 
appropriate  weight  in  source  selection  crite¬ 
ria?  Have  ILS  data  requirements  been 
tailored? 

(5)  Do  contracts  clearly  reflect  baseline 
operational,  maintenance,  and  support  con¬ 
cept  requirements? 

(6)  Is  critical  logistics  support  related 
hardware  and  software  on  contract  or  identi¬ 
fied  for  future  contracts? 

(7)  Is  acquisition  of  engineering  data  on 
contract  and  scheduled  for  delivery  as  part 
of  full-3cale  development  and  production 
contracts? 

b.  Status  of  Documentation.  Con¬ 
siderations  for  this  area  include: 

(1)  Have  training  requirements  been 
projected? 

(2)  Have  software  support  requirements 
been  identified? 

(3)  Have  R&M  performance  criteria  been 
established? 

(4)  Are  R&M  growth  management  plans 
complete?  Have  growth  curves  been  devel¬ 
oped  with  test  schedule  and  reviews  estab¬ 
lished  to  ensure  compliance? 

(5)  Have  the  source  and  level  of  repair 
been  documented? 

(6)  Has  a  cost/benefit  analysis  been 
completed? 

(7)  Are  there  plans  to  produce  test- 
related  documentation  (e.g.,  TEMP,  ORD, 
STAR,  LISP,  MESL,  JRMET  charter)  in  time 
to  influence  OT&E  planning. 

c.  System  Development  and  Maturity. 
Considerations  for  this  area  include: 

(1)  Have  logistics  supportability  goals 
been  established,  and  is  there  a  strategy  for 
achieving  them? 

(2)  Have  plans  been  made  for  mitigating 
the  affect  of  logistics  risks? 

(3)  Are  major  logistics  support-related 
items  (hardware  and  software)  on  track? 

(4)  Are  there  operational  requirements 
which  cannot  be  met  by  the  "as-designed” 
system? 

d.  Programmatic  Voids.  Considerations 
for  this  area  include: 

(1)  Are  depot  support  requirements  valid 
and  keeping  pace  with  program  changes? 

(2)  Is  the  logistics  manpower  to  support 
DPML,  ILSM,  ALC/SPM  fully  supported 
(current  and  projected)? 

(3)  Are  the  depot  manpower  support 
requirements  supported  in  the  budget? 

(4)  Has  an  LSAR  review  team  been 
established? 
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(5)  Is  a  logistics  support  review  process 
in  place? 

(6)  Have  ICS  and  CLS  projections  been 
validated? 

(7)  Has  a  preliminary  list  of  items  for 
contractor  support  been  developed? 

(8)  Have  logistics  risk  drivers  been  re¬ 
examined? 

(9)  Is  the  logistics  support  concept  keep¬ 
ing  pace  with  program  changes  and  reflected 
in  the  budget? 

(10)  Is  the  use  of  computer-aid  acquisi¬ 
tion  and  logistics  support  (CALS)  determined 
and  supported  accordingly? 

(11)  Has  the  need  for  a  Computer  Re¬ 
source  Life  Cycle  Management  Plan 
(CRLCMP)  or  software  support  plan  been 
identified? 

(12)  Have  depot  support  equipment  and 
software  been  identified  and  programmed  to 
support  need  date? 

(13)  Have  tradeoffs  been  completed 
among  design  characteristics,  manpower  skill 
levels,  and  support  concepts? 

(14)  Has  a  product  performance  strategy 
been  developed,  and  is  it  linked  to  logistics 
support  goals.? 

(15)  Is  depot  activation  on  track  with 
need  or  mitigation  plan  in  place? 

e.  Special  Field  Test  Activities.  There 
are  no  considerations  for  this  area.  If  such 
test  activities  are  required,  the  TSG  should 
report  the  test  results. 

f.  Significant  Trends.  If  any  testing 
has  been  done,  the  TSG  should  review  the 
test  information  and  report  any  significant 
trends.  No  rating,  prediction,  or  projection 
is  required. 

5-4.  Milestone  IPA  OAs.  With  early 
involvement  in  the  system’s  acquisition 
program,  and  adequate  OT&E  planning,  an 
OA  for  Milestone  IIIA  may  be  replaced  by 
actual  OT&E  results,  or  even  combined 
DT&E/OT&E  results.  If  an  OA  remains  a 
requirement  for  Milestone  IIIA,  the  preceding 
considerations  should  be  reevaluated  to 
determine  their  applicability  to  this  case.  In 
addition,  content/results  from  previous  OAs 
and  any  lessons  learned  should  be  reviewed 
for  possible  application. 

a.  Planning  Issues.  Considerations  for 
this  area  include: 

(1)  Are  TOs,  training  devices,  and  other 
ILS  elements  needed  for  OT&E  on  contract? 

(2)  Are  test  team  personnel,  test  articles, 
depot  support,  etc.,  fully  programmed  and 
documented  in  the  TPO? 

(3)  Is  training  on  schedule? 

b.  Status  of  Documentation.  Con¬ 
siderations  for  this  area  include: 


(1)  Are  manpo%ver  requirements  docu¬ 
mented  and  supported? 

(2)  Is  postproduction  support  plan  on 
contract  and  funded? 

(3)  Is  the  weapon  system  master  plan  in 
development? 

(4)  Has  the  engineering  data  manage¬ 
ment  plan  been  updated?  Are  data  on  sched¬ 
ule  for  delivery? 

(5)  Has  the  postproduction  support  plan 
been  developed,  and  is  it  on  contract? 

(6)  Have  software  support  plans  or 
CRLCMPs  been  developed? 

(7)  Has  the  PMRT  plan  been  developed? 

(8)  Has  the  Integrated  Spares  Acquisi¬ 
tion  and  Support  Plan  been  developed? 

c.  System  Development  and  Maturity. 
Considerations  for  this  area  include: 

(1)  Are  growth  test  results  used  to  estab¬ 
lish  performance  criteria  in  future  production 
contracts? 

(2)  Are  the  logistics  supportability  goals 
still  rationale? 

(3)  Have  all  major  logistic  risks  been 
mitigated? 

d.  Programmatic  Voids: 

(1)  Are  ICS/CLS  spares  budgets  sup¬ 
ported? 

(2)  Do  the  development  status  and  lead 
times  of  ILS  deliverables  match  need  dates? 

(3)  Is  provisioning  on  contract  with  long 
lead  items  on  or  scheduled  to  be  on  contract 
to  support  operational  need  date? 

(4)  Is  all  equipment  on  contract  or  sched¬ 
uled  to  be  on  contract  to  support  operational 
need  date? 

(5)  Is  depot  activation  on  track  with  the 
program? 

(6)  Is  delivery  of  critical  logistics  support 
items  (hardware  and  software)  on  track? 

(7)  Are  facilities  on  track  to  support 
programmed  need  dates? 

(8)  Is  training  on  schedule? 

e.  Special  Field  Test  Activities.  There 
are  no  considerations  for  this  area.  If  any 
test  activities  have  taken  place,  the  TSG 
should  report  the  results. 

f.  Significant  Trends.  Given  any  test 
results/information,  the  TSG  should  report 
significant  trends  such  as  continued  perform¬ 
ance  discrepancies  with  respect  to  design 
acceptability  (including  R&M),  maintenance 
concepts,  and  support  resource  requirements. 

5-5.  Sample  Operational  Assessments. 
As  with  TPRs,  no  single  OA  will  apply  to  all 
programs  generically.  The  LG  reference 
library  contains  OAs  completed  on  several 
programs.  These  should  be  reviewed  and 
tailored  as  needed  to  satisfy  test  manager 
tasking  for  the  specific  program  in  question. 
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5-6.  Lessons  Learned.  AFOTEC  has  been 
involved  in  OAs  only  since  the  1988  time- 
frame.  Experience  with  them  is  still  evolv¬ 
ing.  However,  some  lessons  learned  are 
noted  here  (undoubtedly  more  will  be  added). 

a.  OAs  are  often  required  concurrently 
with  TPR  activities  described  in  chapter  4. 
As  such,  the  suitability  test  planner  can 
experience  cyclic  intensity  of  work  effort  and 
work  voids.  These  "peaks  and  valleys"  can 
be  minimized  by  using  sound  project  manage¬ 
ment  principles  and  tools.  Also,  designating 
a  backup  or  alternate  suitability  test  planner 
will  contribute  to  balancing  the  work  effort. 

b.  The  suitability  OA  when  first  briefed  to 
the  logistics  staff  should  be  based  on  the 
judgment  of  the  logistics  action  officers 
involved.  As  the  TSG  builds  the  official  OA, 
the  logistics  evaluation  manager  should  strive 
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to  include  those  key  areas  of  the  suitability 
assessment. 

c.  Since  OA  uses  a  color-coded  rating 
scheme,  the  logistics  action  officers  should 
realize  colors  change  for  a  variety  of  reasons. 
Insist  on  a  fair  rating  which  can  be  ade¬ 
quately  substantiated. 

d.  Do  not  let  pride  of  authorship  interfere 
with  the  tasks  at  hand. 

e.  OAs/EOAs  may  or  may  not  require  an 
OA/EOA  plan  or  report  (a  briefing  may 
suffice  for  one  or  both).  APR  55-43  contains 
the  key  assessment  areas  for  which  the 
plan/briefing  will  be  tailored.  The  test 
manager  determines  whether  the  plan  or 
briefing  approach  will  be  used.  The  OA/EOA 
plan  approach  is  more  formal  in  both  content 
and  internal  coordination  requirements. 
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PREPARING  FOR  TEST  EXECUTION 


6-1 


6-1.  General: 

a.  Preparing  for  test  execution  should 
begin  before  TPR  #3  and  requires  frequent 
communication  between  AFOTEC  action 
officers  and  numerous  participating  organiza¬ 
tions.  Acquiring  knowledge  of  these  organi¬ 
zations,  their  role  during  test  execution,  and 
bringing  together  an  actual  test  team  are 
initial  activities.  Once  the  team  is  activated, 
training  the  team  on  both  the  OT&E  and  the 
system  becomes  a  major  focus  for  the  test 
planner.  The  test  team  must  understand 
what  they  are  to  do,  how  to  do  it,  whom 
they  contact  for  additional  guidance,  and 
when  their  jobs  must  be  done.  Ideally,  the 
test  team  will  visit  HQ  AFOTEC  before 
reporting  to  the  test  location  to  meet  their 
TSG  counterparts  and  discuss  one-on-one  all 
specifics  of  the  OT&E. 

b.  Once  prepared,  the  goal  of  the  test 
team  and  TSG  during  test  execution  is  to 
accomplish  the  objectives  specified  in  the  test 
plan.  This  chapter  is  not  meant  to  be  a 
detailed  guide  for  each  test  team  member 
for  all  types  of  systems  that  enter  OT&E. 
Rather,  this  chapter  and  the  next  on  data 
management  provides  general  guidance  on 
generic  topics  common  to  most  OT&E  pro¬ 
grams. 

6-2.  Participating  Organizations: 

a.  Air  Staff.  The  logistics  action  officer 
may  have  contact  with  the  Air  Staff,  the 
program  element  monitor  (PEM),  or  the 
AFOTEC  liaison  officer  (OL-BA)  located  in 
the  Directorate  of  Maintenance  (AF/LGM), 
during  test  execution.  HQ  AFOTEC/OL- 
BA  is  the  primary  suitability  interface  be¬ 
tween  HQ  AFOTEC  and  the  Air  Staff. 

b.  Air  Force  Systems  Command 
(AFSC).  When  acting  as  the  implementing 
command,  AFSC  establishes  a  system  pro¬ 
gram  office  (SPO)  which  manages  the  acqui¬ 
sition  of  the  system  including  development 
test  and  evaluation  (DT&E).  To  conserve 
resources,  AFOTEC  frequently  participates  in 
combined  DT&E/ OT&E.  AFSC  may  provide 
the  test  sites,  test  asset?,  support  equipment, 
and  data  reduction  support  required  for 
combined  testing. 

c.  Air  Force  Logistics  Command 
(AFLC).  As  a  supporting  command,  AFLC 
provides  test  team  personnel,  when  specified 
in  the  test  program  outline  (TPO),  to  aid  in 
conducting  suitability  evaluations.  Through 
the  Air  Force  Acquisition  Logistics  Center 


(AFALC),  AFLC  provides  the  deputy  program 
manager  for  logistics  (DPML)  to  most  SPOs. 

(1)  Air  Logistics  Centers  (ALC).  The 
prime  ALC  provides  reliability  and  main¬ 
tainability  (R&M)  engineers  to  aid  in  con¬ 
ducting  the  R&M  evaluation.  The  prime 
ALC  also  provides  test  team  personnel  to 
conduct  logistics  supportability  evaluations. 
ALC  personnel  needed  to  support  T&E  must 
be  specified  in  the  TPO.  The  prime  ALC,  in 
conjunction  with  the  using  command  and 
based  on  the  software  maintenance  concept, 
will  provide  software  evaluators  to  assess  the 
supportability  of  the  system  software. 

(2)  Deputy  Program  Manager  for  Logistics 
(DPML).  The  DPML,  located  at  the  SPO, 
manages  the  integrated  logistics  support 
(ILS)  program  and  ensures  logistics  issues 
are  adequately  addressed  during  system 
acquisition.  The  DPML  is  an  excellent 
source  of  program-specific  logistics  infor¬ 
mation. 

d.  Using  Command.  The  using  com¬ 
mand  (e.g..  Tactical  Air  Command)  provides 
system  operational  maintenance  personnel  for 
the  test  team  as  specified  in  the  TPO.  The 
using  command  is  the  lead  activity  for  devel¬ 
oping  operational  requirements  against  which 
AFOTEC  evaluates  the  system. 

e.  Air  Training  Command  (ATC).  ATC 
provides  personnel,  when  specified  in  the 
TPO,  to  conduct  system  training  evaluations. 

6-3.  Roles  of  Test  Participants: 

a.  Test  Director.  A  test  director  at  the 
test  location  normally  has  execution  authority 
for  AFOTEC-conducted  OT&E  programs.  The 
test  director  will  be  supported  by  AFOTEC 
headquarters  elements  and  an  AFOTEC 
detachment,  if  applicable.  Within  the  test 
team,  the  deputy  for  logistics  evaluation 
reports  to  the  test  director  and  is  responsible 
for  all  aspects  of  logistics  evaluation.  Simi¬ 
larly,  the  deputy  for  software  evaluation 
reports  to  the  test  director  and  is  responsible 
for  evaluation  of  system  and  support  soft¬ 
ware.  Figure  6-1  depicts  typical  relation¬ 
ships.  The  headquarters  often  provides 
logistics  analysis  support  to  the  test  team  for 
developing  and  running  simulation  models 
and  other  sophisticated  analyses.  Similarly, 
headquarters  software  personnel  may  ad¬ 
minister  software  questionnaires  and  provide 
other  support. 

b.  HQ  AFOTEC: 

(1)  Logistics  Evaluation  Manager.  During 


6-2 


AFOTECP  400-1  15  May  1991 


SUPPORT  RESPONSIBILITIES 


AFOTECP  400-1  15  May  1991 


6-3 


execution  of  test  programs,  the  logistics 
evaluation  manager’s  role  generally  becomes 
that  of  a  test  monitor.  The  logistics  evalua¬ 
tion  manager  maintains  liaison  with  the  test 
team  deputy  for  logistics  evaluation,  deter¬ 
mines  progress  of  the  test,  and  provides 
assistance,  as  required;  maintains  contact 
with  the  using  command  logistics  point  of 
contact,  the  DPML,  and  the  ALC  system/item 
managers,  as  required;  makes  periodic  visits 
to  the  test  site  to  stay  current  with  and 
ensure  proper  progress  of  the  test  activity; 
and  schedules  internal  test  progress  reviews, 
updates  LG  EIS,  and  processes  documents  for 
comment,  as  necessary. 

(2)  Software  Evaluation  Manager  (SEM). 
The  software  evaluation  manager  maintains 
liaison  with  the  test  team  deputy  for  soft¬ 
ware  evaluation,  providing  assistance,  as 
required;  maintains  contact  with  the  using 
command  and  SPO  software  points  of  contact 
and  the  ALC  software  support  manager,  as 
required;  and  makes  periodic  visits  to  the 
test  site  to  administer  software  evaluation 
questionnaires  and  to  monitor  the  progress 
of  the  test.  Before  arrival  of  DSE,  the  SEM 
performs  as  the  DSE.  Also,  the  SEM  is  the 
software  expert  on  the  HQ  AFOTEC  TSG. 

(3)  Logistics  Analysis  Manager.  The 
logistics  analysis  manager  is  the  principal 
logistic  analysis  advisor  to  the  test  support 
group  (TSG)  and  the  test  team.  Analysis 
includes  evaluating  key  program  issues  and 
applying  a  structured  analytical  process  to 
test  design,  execution,  and  reporting.  The 
logistics  analysis  manager  provides  expertise 
in  the  disciplines  of  statistical  inference  and 
sample  sizes,  formulates  mathematical  and 
simulation  models  to  address  critical  opera¬ 
tional  issues  for  OT&E  and  specific  test 
objectives,  and  works  closely  with  the  test 
team  deputy  for  logistics  evaluation  to  ensure 
data  are  properly  collected,  processed,  and 
analyzed. 

c.  Detachment.  Detachments  are  com¬ 
prised  of  several  test  teams  at  a  common 
location  in  order  to  achieve  economies  of 
scale  by  sharing  administrative  support 
functions,  monitor  MAJCOM-conducted 
OT&E,  and  perform  other  activities  as  speci¬ 
fied  in  AFOTECR  55-43,  AFOTEC  Sup  1. 

d.  Test  Team.  The  formal  test  team 
organization  should  be  shown  in  the  test 
plan.  The  test  director  and  the  deputies  for 
logistics  and  software  evaluation  may  aug¬ 
ment  or  refine  the  organization  once  the 
team  is  formed.  The  test  team  conducts 
detailed  test  planning  before  test  start.  After 
the  team  is  established,  detailed  working 
procedures  and  responsibilities  for  team 
members  must  be  determined.  The  test  plan 


and  test  team  procedures  may  change  during 
the  course  of  the  evaluation,  based  on  the 
findings  of  the  test  team  or  changes  to 
critical  issues.  These  changes  may  or  may 
not  be  formal,  but  they  must  be  coordinated 
between  the  test  team  and  HQ  AFOTEC  in 
all  cases. 

(1)  Deputy  for  Logistics  Evaluation 
(DLE).  The  DLE  evaluation  is  responsible 
for  executing  logistics  portions  of  the  test 
plan  and  maintaining  liaison  with  the  HQ 
AFOTEC  logistics  evaluation  manager  using 
the  command  logistics  points  of  contact,  the 
DPML,  and  the  ALC  system  manager. 
Reporting  responsibilities  include  notifying 
the  logistics  evaluation  manager  of  the  status 
of  suitability  testing  and  writing  the  suitabil¬ 
ity  portions  of  the  final  report.  The  deputy 
for  logistics  evaluation  may  also  be  tasked 
with  managing  the  OT&E  aspects  of  the 
service  reporting  system. 

(2)  Deputy  for  Software  Evaluation 
(DSE).  The  DSE  is  responsible  for  executing 
the  software  portions  of  the  test  plan  and 
maintaining  liaison  with  the  HQ  AFOTEC 
software  evaluation  manager,  the  using 
command  and  SPO  software  points  of  contact, 
and  the  software  support  facility  manager. 

(3)  Logistics  and  Software  Evaluators. 
Test  team  logistics  and  software  evaluators, 
regardless  of  their  basic  unit  of  assignment, 
are  under  the  operational  control  of  the  test 
director  and  work  for  the  deputy  for  logistics 
evaluation  or  the  deputy  for  software  evalua¬ 
tion,  respectively.  They  may  be  assigned  to 
the  test  team  on  either  a  temporary  duty  or 
permanent  change  of  station  basis. 

6-4.  Activating  the  Test  Team: 

a.  Local  Support  Relationships.  The 
HQ  AFOTEC  staff  will  generally  have  ar¬ 
ranged  for  resources  to  support  the  test 
before  the  test  team  arrives  using  host-tenant 
support  agreements,  memoranda  of  agree¬ 
ment,  program  introduction  documents,  etc. 
The  test  team  must  carefully  review  these 
documents  to  ensure  all  required  resources 
are  provided. 

b.  Training: 

(1)  Test  team  personnel  should  be  sched¬ 
uled  to  attend  applicable  type  1  training 
(factory  system  training)  en  route  to  their 
test  team  assignment,  if  possible.  If  en  route 
type  1  training  is  not  feasible,  they  should 
attend  as  soon  as  possible  after  reporting  to 
the  team. 

(2)  HQ  AFOTEC  has  a  formal  1-week 
OT&E  course  which  is  appropriate  for  test 
team  management  personnel.  The  deputy  for 
logistics  evaluation,  deputy  for  software 
evaluation,  and  other  selected  senior  logistics 


6-4 


AFOTECP  400-1  15  May  1991 


evaluators  should  schedule  their  attendance 
for  this  course  through  their  test  director, 
AFOTEC/TET,  or  their  logistics  evaluation 
manager. 

(3)  HQ  AFOTEC/LG3  offers  a  series  of 
general  suitability  courses  which  can  be  pre¬ 
sented  at  the  test  team  location  or  at  HQ 
AFOTEC  dependent  on  cost  effectiveness. 
The  topics  available  range  from  a  general 
suitability  overview  to  more  detailed  subjects 
such  as  reliability  growth  analysis  and  nu¬ 
clear  service  reporting.  These  courses  may 
be  requested  and  scheduled  through  the 
logistics  evaluation  manager. 

(4)  Informal  training  and  indoctrination 
will  be  provided  to  the  test  team  by  the 
headquarters  logistics  evaluation  manager, 
analyst,  and  software  evaluator,  as  required. 
This  indoctrination  should: 

(a)  Acquaint  the  logistics  team  members 
with  the  background  of  the  test  program,  the 
test  schedule,  and  the  key  milestones  of  the 
test  in  relation  to  major  decision  points. 

(b)  Provide  a  clear  understanding  of  the 
test  purpose,  the  test  objectives,  the  defined 
operational  and  maintenance  concepts,  and 
the  basic  concepts  of  operational  suitability 
test  and  evaluation. 

(c)  Ensure  test  team  members  under¬ 
stand  the  organization  of  the  test  team,  the 
chain  of  command,  their  role  on  the  team, 
and  to  whom  they  are  responsible. 

(d)  Ensure  test  team  members  under 
stand  the  procedures  for  interaction  and 
coordination  with  other  agencies,  organiza¬ 
tions,  and  commands. 

(e)  Ensure  test  team  members  under¬ 
stand  the  data  management  plan  and  their 
responsibilities  in  its  execution.  They  must 
understand  their  special  role  as  evaluators  as 
well  as  maintainers  and  should  dry-run  data- 
related  activities  such  as  joint  reliability  and 
maintainability  evaluation  teams  (JRMET) 
and  SR  prioritization  boards  before  test  start. 

(f)  Ensure  test  team  members  under¬ 
stand  the  meaning  of  restricted  handling  of 
source  selection  sensitive  data  LAW  AFR 
70-15,  Formal  Source  Selection  for  Major  Ac¬ 
quisitions. 

(g)  Ensure  test  members  understand 
security  aspects  of  the  program. 

6-5.  Review  of  Program  Documenta¬ 
tion: 

a.  Test  Plan.  The  test  team  should 
review  the  test  plan  (if  possible,  in  conjunc¬ 
tion  with  the  TSG)  to  acquaint  themselves 
with  the  critical  issues,  test  objectives,  and 
associated  measures  of  effectiveness.  They 
should  review  test  scenarios  and  methods  to 
ensure  they  can  accomplish  the  objectives. 


The  test  plan  should  be  viewed  as  a  living 
document.  If  a  change  is  required  to  the 
test  plan,  the  test  team  should  contact  the 
test  manager.  Test  plan  changes  must  be 
thoroughly  coordinated  and  approved  by  HQ 
AFOTEC.  The  test  team  does  not  have  the 
authority  to  unilaterally  change  the  test  plan. 

b.  Test  Program  Outline  (TPO).  The 
test  team  should  review  the  TPO  to  ensure 
sufficient  resources  have  been  requested  to 
accomplish  the  test  and  the  resources  have 
been  or  will  be  acquired. 

c.  Contract.  Review  of  the  contract  may 
provide  the  deputy  for  logistics  evaluation 
insight  into  the  data  system  which  will  be 
used  in  the  test.  Of  particular  concern  are 
limitations  that  may  occur  because  of  either 
contractual  requirements  or  omissions  of 
requirements  which  are  required  to  support 
OT&E. 

d.  System  Operational  and  Mainte¬ 
nance  Concepts.  The  test  team  should  re¬ 
view  these  documents  to  provide  insight 
when  evaluating  and  interpreting  the  test 
results. 

e.  Threat  Analysis.  Review  of  the 
threat  analysis  should  provide  insight  into 
the  adequacy  of  the  test  scenario.  The 
scenario  or  test  events  may  need  to  be  modi¬ 
fied  to  address  logistical  efforts  within  the 
threat  environment,  such  as  operating  in 
chemical  protective  gear. 

£  Integrated  Logistics  Support  Plan 
(ILSP).  The  test  team  should  review  the 
ILSP  to  acquaint  themselves  with  program 
acquisition  logistics  requirements  and  strat¬ 
egy.  This  information  will  aid  in  under¬ 
standing  the  test  environment  and  in  eval¬ 
uating  test  results. 

g.  Other  Documents.  The  following 
documents  provide  further  insight  into  the 
program  and  test  environments: 

(1)  Program  management  directive 
(PMD). 

(2)  Program  management  plan  (PMP). 

(3)  Mission  need  statement  (MNS). 

(4)  Operational  Requirements  Document 
(ORD),  and  requirements  correlation  matrix 
(RCM). 

(5)  Test  and  evaluation  master  plan 
(TEMP). 

(6)  Decision  coordinating  paper  (DCP). 

h.  Document  Revisions.  The  DLE/DSE 
should  notify  the  logistics  evaluation  manager 
of  any  suitability-related  inconsistencies, 
inaccuracies,  and  testing  problems  discovered 
during  the  document  review  process.  If  the 
logistics  evaluation  manager  cannot  solve  the 
problem,  the  test  manager,  in  concert  with 
the  TSG  and  test  team,  will  resolve  these 
issues. 
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(3)  Establish  name  and  face-to-face  con¬ 
tact  with  OT&E  representatives  in  the  using 
command,  the  supporting  commands,  the 
ALCs,  the  DPML,  and  above  all,  in  your  own 
test  team. 

(4)  Attend  program  reviews,  working 
groups,  and  integrated  logistics  support 
management  team  (ILSMT)  meetings  to  en¬ 
sure  OT&E  issues  are  adequately  addressed 
and  to  remain  informed  of  program  status 
and  planning. 

(5)  Coordinate  inputs  to  the  agenda  of 
any  of  the  above  meetings  to  ensure  a  thor¬ 
ough  understanding  of  the  AFOTEC  position. 

(6)  Keep  each  other  (logistics  evaluation 
manager  and  deputy  for  logistics  evaluation) 
informed  of  progress  and  problems. 

(7)  Maintain  a  contact  file  with  relevant 
information  about  the  contact:  person,  place, 
subject,  problem  nature,  when  contacted, 
results,  further  action  required,  etc. 

b.  Preparation  for  Test  Execution: 

(1)  Logistics  evaluation  managers,  ana¬ 
lysts,  and  software  evaluation  managers  must 
take  the  lead  in  ensuring  the  DLE,  DSE,  and 
others  are  aware  of  and  receive  available 
OT&E  and  operational  suitability  training. 

(2)  The  test  team  deputies  should  each 
assemble  their  personnel,  analyze  the  test 
plan,  determine  the  most  effective  manner  to 
accomplish  the  test,  and  create/expand  de¬ 
tailed  test  procedures.  Using  a  copy  of  the 
test  schedule,  ensure  test  events  are  tied  to 
specific  test  objectives. 

(3)  Test  team  deputies  should  avoid 
becoming  too  involved  in  day-to-day  details 
of  test  support  to  the  detriment  of  managing 
their  overall  test  and  evaluation  require¬ 
ments.  Avoid  becoming  an  action  officer. 
The  DLE  should  not  become  the  test  supply 
or  mobility  officer  at  the  expense  of  the 
OT&E  mission.  The  DSE  is  not  the  Word¬ 
Perfect  expert  or  Z-248  guru  on  the  test 
team. 

(4)  Keep  the  interfaces  working  between 
the  test  team  and  HQ  AFOTEC. 

(5)  Test  teams  should  be  given  a  great 
deal  of  flexibility  in  achieving  test  objectives. 

(6)  Agreements  to  provide  test  support, 
e.g.,  with  the  host  base  commander,  should 
be  in  writing. 

(7)  Contractor  data  may  not  be  complete 
or  may  be  in  a  format  unfamiliar  to  the 
logistics  evaluators.  The  test  team  must 
identify  and  compensate  for  differences  in 
definitions  in  the  OT&E  data  base. 

(8)  Discussions  with  the  contractor  can  be 
extremely  sensitive.  The  test  team  should 
always  provide  advance  notification  to  the 
SPO  of  any  involvement  with  the  contractor. 
AFOTEC  has  no  authority  to  direct  contrac¬ 


tors. 

^9)  The  best  single  source  of  information 
on  testing  is  AFOTEC  personnel  who  have 
done  it  before.  No  formal  data  source  or 
training  program  can  provide  as  much  guid¬ 
ance  as  experienced  personnel. 

(10)  Bring  the  test  team  deputies  on 
board  early,  preferably  for  final  review  of 
the  test  plan  at  HQ  AFOTEC.  Follow  them 
with  a  small  cadre  (e.g.,  maintenance  officer 
and  data  collectors)  well  in  advance  of  test 
execution.  Get  the  whole  test  team  in  place 
and  trained  by  the  start  of  test.  However, 
keep  abreast  of  test  schedule  slips,  and  do 
not  have  the  bulk  of  the  test  team  on  board 
too  early. 

(11)  The  test  team  should  dry  run  the 
JRMET  and  SR  prioritization  boards  (at  least 
30  days  before  test  start)  to  mitigate  inevi¬ 
table  problems  with  board  procedures  and 
data  review. 

(12)  Bases  operationally  oriented  will 
have  a  different  perception  of  the  test  than 
bases  where  tests  are  routinely  conducted. 
The  test  team  must  explain  the  mission, 
emphasis,  and  procedures  of  OT&E  to  base 
managers. 

(13)  Test  team  management  should  re¬ 
view  applicable  lessons  learned  from  the 
OT&E  Lessons  Learned  Data  Bank.  Do  not 
repeat  the  mistakes  of  your  predecessors,  but 
capitalize  on  their  observations. 

c.  Active  Testing: 

(1)  Data  collection  requires  much  disci¬ 
pline  within  the  test  team  to  collect  com¬ 
plete,  accurate  suitability  data. 

(2)  Test  team  coordination  with  the  con¬ 
tractor’s  maintenance  activity  will  be  more 
difficult  if  the  contractor  has  no  central 
maintenance  control  office.  Collection  of 
quantitative  and  qualitative  maintenance 
data  will  require  the  test  team  to  coordinate 
with  individual  maintenance  shops. 

(3)  The  DLE  may  be  able  to  accomplish 
the  suitability  evaluation  easier  if  his  or  her 
office  is  physically  close  to  the  maintenance 
activity. 

(4)  Careful  quality  control  is  required  in 
data  processing  to  ensure  data  are  not  lost 
during  input  or  sorting.  Compare  sample 
size  at  input  and  output. 

(5)  Differences  between  the  test  environ¬ 
ment  and  operational  environment  require 
careful  subjective  analysis.  Rationale  used 
to  extrapolate  test  environment  results  to  the 
operational  environment  should  be  thoroughly 
documented  in  the  detailed  test  procedures. 

(6)  Maintenance  personnel  filling  a  dual 
role  of  equipment  maintainers  and  logistics 
evaluators  tend  to  forget  test  priorities  and 
acquire  a  vested  interest  in  making  the  test 
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6-6.  Key  Meetings: 

a.  Program  Meetings.  During  the 
OT&E  planning  process,  members  of  the  TSG 
attend  and  participate  in  program-related 
meetings  (e.g.,  test  planning  working  group 
(TPWG)  meetings,  program  design  reviews, 
and  ILSMT  meetings).  However,  after  the 
test  team  has  been  activated,  test  team 
members  should  also  attend  these  meetings. 
The  test  team  should  participate  in  program- 
related  meetings  to  ensure  they  are  fully 
informed  of  program  status  and  planning. 

b.  OT&E  Meetings.  Aside  from  meet¬ 
ings  scheduled  by  the  test  director,  test  team 
and  suitability  TSG  members  may  partici¬ 
pate  in  JRMETS  and  materiel  improvement 
project  review  boards  (MIPRB).  JRMETS 
are  discussed  in  chapter  7.  MIPRBs  are  dis¬ 
cussed  in  paragraph  6-8. 

6-7.  Active  Testing: 

a.  Test  members  should  direct  activities 
during  active  testing  toward  achieving  test 
objectives  and  producing  a  meaningful  final 
report.  Place  emphasis  on  collecting  and 
analyzing  data  required  by  the  test  plan  and 
the  detailed  test  procedures.  The  key  to 
success  is  review  and  analysis  of  data  as 
they  are  collected,  particularly  during  long 
tests.  Without  this  review,  problems  in  the 
data  collection  process  may  be  overlooked 
until  it  is  too  late  to  recover  and  major  test 
objectives  become  unachievable. 

b.  Operational  suitability  data  may  be 
divided  into  two  categories — quantitative  and 
qualitative.  The  collection  of  quantitative 
operational  suitability  data  is  frequently  tied 
to  operational  effectiveness  test  events.  As 
a  result,  logistics  evaluators  must  work 
closely  with  effectiveness  evaluators  to  ensure 
logistics  data  are  collected  on  all  test  events. 
Collection  must  be  followed  by  a  detailed 
audit  to  ensure  all  the  required  data  were 
collected  and  correct  coding  was  used.  Anal¬ 
ysis  of  the  data  should  be  accomplished  by 
logistics  evaluators  and  the  JRMET.  Calcu¬ 
lation  of  measures  of  effectiveness  should 
occur  as  early  as  possible  to  verify  the  data 
collection  and  reduction  processes  and  allow 
early  correction  of  inadequacies.  Additional¬ 
ly,  MOEs  should  be  updated  periodically  to 
identify  trends  and  areas  of  concern  and  to 
ensure  the  continued  adequacy  of  the  data 
collecting  process.  Test  data  analysis  should 
also  begin  as  early  as  possible  and  continue 
throughout  the  test.  This  analysis  should  be 
aimed  at  identifying  trends  and  deficiencies 
and  at  determining  their  causes. 

c.  The  collection  of  qualitative  operational 
suitability  data  must  be  carefully  planned 
and  executed  to  ensure  all  data  are  collected 


and  analyzed  before  the  test  is  over.  Quali¬ 
tative  data  collection  involves  a  wide  range 
of  activities  which  vary  from  the  completion 
of  questionnaires  to  the  review  of  program 
office  logistics  planning  documents.  Regard¬ 
less  of  sources,  qualitative  data  may  be 
thought  of  as  opinions  of  one  or  more  in¬ 
dividuals.  As  such,  care  must  be  taken  to 
ensure  objectivity  is  maintained  throughout 
the  process  of  collecting  and  analyzing  these 
data.  While  some  subjective  analyses  may  be 
conducted  toward  the  end  of  testing,  they 
should  be  done  in  time  to  allow  a  detailed 
analysis  of  significant  findings. 

6-8.  Service  Reporting  (SR): 

a.  A  primary  OT&E  function  is  to  iden¬ 
tify  and  report  deficiencies  that  impact  the 
operational  suitability  and  operational  effec¬ 
tiveness  of  the  weapon  system.  Early  identi¬ 
fication  of  deficiencies  provides  the  opportu¬ 
nity  to  fix  problems  at  a  lower  cost. 

b.  The  test  team  must  develop  rigorous 
procedures  to  identify,  verify,  report,  and 
track  deficiencies.  Technical  Order  (TO)  00- 
35D-54,  USAF  Material  Deficiency  Reporting 
and  Investigating  System,  and  AFR  55-43, 
provide  guidance  for  establishing  these  proce¬ 
dures.  Deficiencies  reported  should  not  be 
limited  to  items  for  which  the  contractor  is 
responsible.  For  example,  if  the  system 
being  evaluated  consists  of  government-fur¬ 
nished  equipment  (GFE)  and  contractor- 
developed  equipment  and  a  deficiency  is 
discovered  that  relates  to  the  GFE,  the  test 
team  should  submit  a  service  report. 

c.  The  test  director  is  responsible  for  the 
SR  program  and  identifies  who  will  perform 
duties  related  to  monitoring  all  SR  activities. 
The  test  team  deputy  for  logistics  evaluation 
may  be  tasked  with  handling  the  administra¬ 
tion  of  the  service  reporting  process  including 
representing  the  Operational  Test  Agency 
(OTA)  on  MIPRBs  and  chairing  SR  prioritiza¬ 
tion  boards.  MIPRBs  assign  funding  to  the 
SRs.  Close  coordination  with  all  affected 
agencies  is  required  throughout  the  test. 
The  test  team  should  notify  the  program 
office  and,  if  warranted,  the  contractor  of 
identified  and  suspected  system  deficiencies. 

6-9.  Lessons  Learned: 

a.  Working  Relationships.  Many  things 
can  be  done  to  ensure  effective  working 
relationships.  Among  these  are: 

(1)  Numbers  and  types  of  participating 
organization  personnel  necessary  to  support 
OT&E  and  the  periods  of  time  they  are 
required  must  be  specified  in  the  TPO. 

(2)  Determine  the  formal  and  informal 
lines  of  authority  in  the  program. 
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item  operate.  This  can  skew  test  data, 
especially  where  testers  have  higher  skill 
levels  than  planned  for  the  operational 
environment. 

(7)  Site  Activation  Task  Force  (SATAF). 
SATAF  activity  is  not  formally  related  to 
OT&E;  however,  information  gathered  during 
OT&E  can  be  of  assistance  to  a  SATAF. 
AFOTEC  may  render  assistance  to  SATAFs 
by  placing  them  on  the  information  list  for 
SRs  and  other  test  reports  that  might  be  of 
interest.  AFOTEC  can  provide  additional 
assistance  by  answering  questions  which  may 
arise  at  SATAF  meetings. 

(8)  AFOTEC/LG  action  officers  should 
maintain  frequent  (daily)  contact  with  the 
test  team.  Items  of  interest  should  be  for¬ 
warded  to  the  director  of  logistics,  depend¬ 
ing  on  urgency.  Timely  submission  of  trip 
reports  is  a  must. 

d.  Service  Reporting: 

(1)  Transmit  only  those  deficiencies  which 
are  thoroughly  screened  and  validated  by  the 
team  and  test  director.  Inaccurate,  trivial, 
or  improperly  documented  SRs  degrade  the 
team’s  credibility  and  needlessly  congest  the 
SR  pipeline. 

(2)  Track  every  SR  until  the  reported 
deficiency  is  resolved  or  the  test  is  over. 
Establishment  of  material  improvement 
project  (MIP)  by  the  SPO  or  the  subsequent 
establishment  and  funding  of  an  engineering 
change  proposal  (ECP)  are  not  sufficient 
grounds  to  close  the  books  on  an  SR.  SRs 
must  not  be  closed  until  a  fix  has  been 
implemented  and  verified  or  the  MIP  review 
board  unanimously  agrees  on  closure. 

(3)  Consider  using  an  automated  test 
team  SR  tracking  system  for  large  programs. 
In  one  test,  the  SPO  loaded  MIPs  into  an 
independent  computer  file  to  which  the  test 
team  had  access.  HQ  AFOTEC/LG  has  a 
microcomputer-based  SR  tracking  program 
available  to  generate  and  track  test  team- 


initiated  SRs. 

(4)  The  SR  system  requires  many  man¬ 
hours.  Do  not  underestimate  the  workload. 

(5)  Do  not  hold  SRs  and  report  them  all 
at  the  end  of  the  test.  Deficiencies  should 
be  identified  in  a  timely  manner  for  correc¬ 
tive  action.  Timelines  for  submission  and 
processing  of  SRs  are  specified  in  TO  00- 
35D-54. 

(6)  Do  not  assume  deficiencies  identified 
on  a  system  still  under  a  reliability  improve¬ 
ment  warranty  will  be  corrected  'free'’  by 
the  contractor.  Investigate  this  peculiarity 
carefully  with  the  SPO  and  report  any  vali¬ 
dated  deficiency  as  an  SR.  Procedures  for 
SR  documentation  of  warranted  items  are 
specified  in  TO  00-35D-54. 

(7)  Be  thorough  in  documenting  the  SR. 
The  SPO  action  officer  may  not  be  familiar 
with  the  problem.  Send  pictures  or  videotape 
if  necessary.  Be  prepared  to  provide  an 
exhibit  or  sample  of  failed,  poor  workman¬ 
ship,  or  nonconforming  items. 

(8)  Work  SRs  before,  during,  and  after 
testing.  Get  started  early.  Appoint  an  SR 
monitor  on  day  one. 

(9)  Make  sure  everyone  on  the  team 
knows  (before  test  execution)  the  full  details 
of  the  SR  program,  including  filling  out 
forms;  researching  the  contract  and  part 
numbers,  nomenclature,  classification,  down¬ 
grade  instructions,  etc. 

(10)  Establish  and  widely  publicize 
ground  rules  for  resolving  differences  of 
opinion  on  SR  closures.  SRs  may  not  be 
unilaterally  closed  by  the  SPO  or  others  if 
there  are  objections. 

(11)  Include  SRs  as  an  item  for  mission 
debrief/lessons  learned. 

( 12)  Give  inputs  to  an  assigned  SR  writer 
for  the  affected  functional  area.  SR  writers 
should  (within  a  day  or  two,  while  memories 
are  fresh)  revise  the  initial  version  of  the 
draft  SR,  if  required,  based  on  feedback  from 
crews,  data  collectors,  etc. 
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Chapter  7 

MANAGING  OPERATIONAL  SUITABILITY  DATA 


7-1.  General: 

a.  Data  management  is  the  process  of 
identifying,  collecting,  processing,  and  dispos¬ 
ing  test-related  data.  The  purpose  of  this 
chapter  is  to  familiarize  T&E  personnel  with 
data  management  and  analysis,  data  collec¬ 
tion  and  control,  and  common  modeling 
inputs  and  responsibilities.  It  also  provides 
a  discussion  of  the  logistics  support  analysis 
(LSA)  program. 

b.  An  AFOTEC  test  team  uses  TSG/test 
team-developed  methods  for  data  manage¬ 
ment  based  on  the  test  plan,  test  team 
resources,  and  the  dictates  of  ongoing  pro¬ 
gram  events.  OT&E  test  plans  are  reviewed 
to  identify  required  data.  It  should  then  be 
determined  whether  the  data  will  be  collected 
by  the  test  team  or  from  the  tests  and 
activities  conducted  and  reported  by  other 
organizations  (e.g.,  contractors,  development 
testers.)  If  the  decision  is  to  collect  the 
required  data  from  the  tests  conducted  by 
others,  testing  and  reporting  plans  of  those 
agencies  should  be  reviewed  to  determine 
whether  data  requirements  will  be  satisfied 
with  or  without  AFOTEC  test  team  participa¬ 
tion.  If  it  appears  the  data  requirements 
will  be  satisfied,  necessary  coordination  must 
be  accomplished  and  the  data  collected  and 
analyzed.  Unless  problem  areas  are  iden¬ 
tified  which  require  analysis  of  raw  test  data, 
it  is  acceptable  to  use  test  and  analysis 
reports  from  other  agencies  at  the  highest 
possible  level  of  aggregation.  If  test  team 
participation  is  required,  they  should  develop 
and  coordinate  necessary  plans  and  collect 
and  analyze  data.  In  those  cases  where  data 
requirements  cannot  be  satisfied,  the  test 
plan  should  be  modified  or  data  requirements 
reassessed. 

c.  AFR  800-18  charges  the  implementing 
command  (normally  Air  Force  Systems  Com¬ 
mand  (AFSC))  with  the  responsibility  of 
implementing  data  collection  for  measuring 
and  evaluating  R&M  during  development 
test  and  evaluation  (DT&E)  or  OT&E  pro¬ 
grams  and  with  establishing  a  joint  reliabil¬ 
ity  and  maintainability  evaluation  team 
(JRMET)  for  major  system  acquisitions. 

d.  Suitability  test  personnel  should  nor¬ 
mally  use  the  following  procedures: 

(1)  Specify  the  R&M  data  base  in  the 
AFOTEC  test  plan.  The  data  base  systems 
below  normally  apply: 

(a)  Edwards  APB  SEDS.  OT&E  pro¬ 
grams  supported  by  Edwards  AFB  R&M  data 


processing  use  the  SEDS  maintained  by 
AFFTC/ENAR  or  6520  Test  Group. 

(b)  Eglin  AFB  SEDS.  OT&E  programs 
supported  by  Eglin  AFB  R&E  data  processing 
use  the  SEDS  R&M  data  system  maintained 
by  Det  24,  ASD/ENP. 

(c)  AFOTEC  OMNIVORE.  OT&E 
programs  not  supported  by  the  Edwards  or 
Eglin  SEDS  use  AFOTEC’s  OMNIVORE/ 
MICRO-OMNTVORE  computer  programs. 

(d)  Manually  kept  logs  and  videotape 
libraries. 

(2)  Requirements  for  computer  time, 
programmer  support,  video  recorders,  instru¬ 
mentation,  and  similar  resources  must  be 
included  in  the  test  program  outline  (TPO). 

(3)  Unique  operational  suitability  data 
system  requirements  will  be  coordinated  with 
HQ  AFOTEC/TE  through  HQ  AFOTEC/LG  to 
e  ns  tire  the  requirement  is  not  met  by  current 
capabilities,  new  requirements  comply  with 
R&M  standards,  and  new  developments 
receive  wide  dissemination  and  appropriate 
use. 

(4)  Unique  automated  data  processing 
requirements,  such  as  data  processing  equip¬ 
ment  or  contractor  support,  should  be  coordi¬ 
nated  with  HQ  AFOTEC/RM  through  HQ 
AFOTEC/LG 

(5)  Test  teams  must  participate  on  the 
JRMET.  Normally,  the  primary  representa¬ 
tive  is  the  deputy  for  logistics  evaluation, 
assisted  by  the  deputy  for  software  evalua¬ 
tion,  the  R&M  analyst/engineer,  and  mainte¬ 
nance  personnel.  While  AFOTEC  is  not 
bound  by  the  JRMETs  interpretation  of  the 
data,  agreement  is  usually  reached  and  a 
common  data  base  is  established  for  use  by 
all  participating  agencies.  In  cases  where 
agreement  is  not  reached,  the  specific  items 
should  be  coded  in  the  common  R&M  data 
base  for  separate  DT&E  and  OT&E  analysis. 

7-2.  Data  Management  and  Analysis 
Plan  (DMAP).  Reference  AFR  55-43,  for 
additional  guidance. 

a.  The  DMAP,  which  parallels  the  test 
plan  in  development,  is  the  primary  tool  to 
ensure  required  data  are  identified,  recorded, 
collected,  reduced,  processed,  verified,  ana¬ 
lyzed,  and  evaluated  to  support  each  test 
objective.  It  is  developed  jointly  by  the  TSG 
and  test  team.  The  DMAP  is  designed  to  be 
a  working  document  used  in  the  actual 
conduct  of  a  test  and  is  not  intended  for 
external  coordination.  It  must  provide  a 
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flexible  means  of  updating  test  data  require¬ 
ments  and  test  philosophy  to  meet  the  needs 
of  the  test  program.  It  should  address: 

(1)  Dissemination  of  data  between  dif¬ 
ferent  locations. 

(2)  Data  processes  to  avoid  duplication  of 
effort. 

(3)  Focal  points  and  responsibility  centers 
for  data  management. 

(4)  Who  will  collect  the  data,  what  they 
are  to  do  with  them,  where  they  go,  what 
will  be  done  next,  etc. 

(5)  Calculations  and  data  processing 
equipment  to  be  used.  The  analysis  tech¬ 
niques  and  evaluation  procedures  should  spell 
out  who  will  do  what  with  which  technique. 

(6)  Disposition  policies  for  all  test  data 
describing  the  exact  procedures  and  media  to 
be  used  in  storage. 

b.  The  body  of  the  DMAP  is  generally 
organized  by  test  objectives,  but  for  a  com¬ 
plex  test  it  may  be  organized  by  test  phases 
or  test  environment.  As  an  extreme  ex¬ 
ample,  a  multiservice  test  may  require  the 
collection  of  data  at  different  test  sites, 
transportation  to  a  control  data  processing 
agency,  and  extensive  quality  control  to 
ensure  compatibility.  These  requirements 
may  dictate  the  need  for  a  separate  data 
collection  plan  coordinated  through  partici¬ 
pating  agencies. 

c.  The  DMAP  generally  begins  with  a 
synopsis  of  the  test,  description  of  the  test 
articles,  test  environment  and  scenario  test 
objectives,  models  which  may  be  used  for 
analysis,  contractor  involvement,  and  any 
other  factors  which  are  expected  to  impact 
data  management.  Some  estimates  of  the 
volume  and  diversity  of  the  test  data  must 
be  developed  as  a  starting  point  for  data 
management  because  these  will  affect  the 
choice  of  data  systems. 

d.  The  DMAP  should  address  data  re¬ 
quirements  which  differ  from  common  prac¬ 
tice  or  require  special  attention  from  the  test 
team.  An  operational  suitability  evaluation 
generally  uses  established  data  system, 
procedures,  and  definitions,  e.g.,  SEDS  or 
core  automated  maintenance  system  (CAMS). 
However,  some  test  objectives,  such  as  dor¬ 
mant  reliability  or  built-in  test  (BIT)  effec¬ 
tiveness,  may  require  nonstandard  data 
items.  The  DMAP  should  specify  in  greater 
detail  the  data  requirements,  procedures,  and 
definitions  for  these  objectives.  The  DMAP 
should  also  address  any  data  collection 
requirements  unique  to  the  test. 

e.  Special  attention  must  be  given  to  the 
use  of  contractor  data.  Data  requirements 
must  be  coordinated  with  the  SPO  at  the 
earliest  possible  date  to  ensure  they  are 


specified  by  contract.  Where  possible,  devel¬ 
op  specific  techniques  to  ensure  the  contrac¬ 
tor  data  are  representative  of  actual  condi¬ 
tions. 

f.  The  DMAP  should  include  provisions  for 
maintaining  the  data  system.  Experience 
from  several  previous  test  programs  has 
shown  operational  data  processing  packages 
do  not  always  work  as  prescribed.  Additional 
programming  support  may  be  required  to 
satisfy  the  MOEs.  Other  test  programs  have 
generated  additional  test  objectives  as  the 
program  progressed.  These  new  require¬ 
ments  may  also  need  programming  support. 
Latent  errors  may  surface  in  a  data  analysis 
program  which  has  been  operational  for  an 
extended  period.  Programming  support  for 
data  system  maintenance  is  not  generally 
available  within  a  test  team;  therefore,  the 
support  must  be  obtained  from  another 
agency.  Programming  support  should  con¬ 
tinue  throughout  the  test  program  if  the  data 
processing  system  is  contractor  supplied.  For 
any  specially  developed  data  programs, 
system  maintenance  requirements  should 
receive  special  attention  and  should  be  spec¬ 
ified  in  detail  in  the  DMAP. 

g.  The  DMAP  should  carefully  document 
intended  service  reporting  (SR)  procedures. 
SR  screening  points,  exhibit  holding  areas, 
prioritization  methods,  prioritization  board 
membership,  etc.,  must  be  carefully  thought 
out  and  specified.  If  the  deficiency  and 
enhancement  analysis  and  ranking  technique 
(DART)  is  to  be  used  to  prioritize  SRs,  rating 
areas  and  weights  must  be  formulated  and 
coordinated  with  the  using  command  prior  to 
test  start.  Coordinated  rating  areas,  weights, 
DART  forms,  and  prioritization  board  proce¬ 
dures  should  be  specified  in  the  DMAP.  A 
"DMAP  dry  run"  should  be  conducted  jointly 
by  the  TSG  and  test  team  before  the  test 
readiness  briefing. 

7-3.  Air  Force  Data  Systems: 

a.  Introduction-  This  section  discusses 
automated  data  systems  that  may  support 
suitability  OT&E.  Data  collection  is  only  the 
first  step.  The  test  team  must  then  collate 
and  evaluate  the  data.  For  major  programs 
with  large  quantities  of  data,  use  of  auto¬ 
mated  systems  may  reduce  the  amount  of 
manual  analysis  of  test  data  required.  Some 
current  data  systems  are  as  follows: 

(1)  Core  Automated  Maintenance  System 
(CAMS).  CAMS  is  the  Air  Force’s  standard 
system  for  collecting  maintenance  data  for 
operational  Air  Force  systems.  CAMS  was 
primarily  designed  to  support  base-level 
maintenance  managers  in  comparing  perform¬ 
ance  of  equipment  and  personnel  with  sched- 
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uled  requirements  and  in  identifying  trends. 
Output  products  from  CAMS  are  used  in 
supporting  decisions  on  equipment  modifica¬ 
tions  and  provisioning.  Products  are  also 
available  to  provide  quantitative  data  for 
failures/discrepancies,  maintenance  actions, 
and  maintenance  man-hours  for  a  specific 
calendar  period. 

(2)  System  Effectiveness  Data  System 
(SEDS).  SEDS  is  an  AFSC  reliability  and 
maintainability  data  acquisition,  storage, 
retrieval,  reporting,  and  analysis  system 
developed  primarily  to  be  used  for  test  pur¬ 
poses. 

(a)  There  are  two  primary  versions  of 
SEDS.  One  is  maintained  by  Air  Force 
Flight  Test  Center  (AFFTC),  the  other  by  Det 
24,  ASD/ENP.  Both  use  AFSC  Forms  258, 
Maintenance  Discrepancy/Production  Credit 
Record,  and  258-4  (four-part  copy).  The 
ASD/ENP  SEDS  is  a  version  of  AFFTC  SEDS 
developed  inhouse  by  ASD/ENP,  Eglin  AFB. 
It  is  managed  by  the  Directorate  of  Manufac¬ 
turing  and  Supportabilitv.  A  major  dif¬ 
ference  in  the  ASD/ENP  version  of  SEDS  is 
it  can  track  system  or  subsystem  operating 
time  independent  of  aircraft  time  to  conduct 
reliability  analyses.  ASD/ENP  SEDS  can 
also  be  used  to  conduct  reliability  growth 
analyses  routine.  Time  to  failure  is  tracked 
in  the  ASD/ENP  version  of  SEDS  but  not  in 
the  AFFTC  version. 

(b)  SEDS  has  many  common  elements 
with  the  CAMS,  but  it  is  more  useful  be¬ 
cause  it  provides  a  capability  for  a  narrative 
presentation  of  discrepancies  and  corrective 
actions,  delay  codes,  required  Air  Force 
specialty  codes,  ground  support  equipment 
data,  diagnostic  data,  and  technical  order 
data.  During  OT&E,  AFOTEC  has  a  need  to 
obtain  extensive  information  on  a  system  in 
order  to  determine  the  degree  to  which  it  can 
be  placed  into  field  use.  Once  fielded,  some 
data  elements  are  not  necessary  and  hence 
not  tracked  by  CAMS. 

(3)  Maintenance  Management  Informa¬ 
tion  and  Control  System  (MMICS): 

(a)  MMICS  was  designed  to  provide 
real-time  management  information  to  base- 
level  maintenance  managers.  MMICS  in¬ 
cludes  a  wide  variety  of  control  and  report¬ 
ing  routines  to  reduce  the  need  to  prepare 
management  documentation  manually.  Air 
Force  Manual  66-278,  MMICS  Users  Manual, 
volumes  1  through  3,  with  the  implementing 
directives,  provides  complete  instructions  for 
MMICS  use  and  descriptions  of  the  output 
products.  For  the  purposes  of  logistics  eval¬ 
uation  during  test  and  evaluation,  the  pri¬ 
mary  MMICS  functions  of  interest  are  the 
on-line  reporting  of  equipment  inventory  and 


status  fully  mission  capable  'FMC).  partially 
mission  capable  <PMC),  and  not  mission 
capable  (NMC).  This  function  permits  input 
to  the  computer  of  inventory  and  status 
changes  a3  they  occur  by  means  of  terminals 
located  in  the  maintenance  complex.  The 
output  product  (Aerospace  Vehicle  Status 
Report)  provides  a  status  summary  report  by 
reason  and  cause  for  a  period  of  99  days, 
status  distribution  by  day,  and  average 
distribution  time  for  31  days,  or  a  combina¬ 
tion  of  status  summary  and  distribution  of  31 
days. 

(b)  In  addition  to  the  Aerospace  Vehicle 
Status  Report,  MMICS  contains  segments  to 
manage  time  compliance  technical  orders, 
mechanized  equipment  records,  job  standards, 
work  unit  codes,  operational  events,  delayed 
discrepancy  file,  maintenance  administration, 
and  training. 

(4)  Equipment  Inventory,  Status,  and 
Utilization  Reporting  Systems.  AFR  65-110. 
Aerospace  Vehicle  and  Equipment  Inventory. 
Status,  and  Utilization  Reporting  System 
(AVISURS),  forms  the  basis  for  developing 
the  Air  Force  programming  documents  and 
related  budgets  and  manning  requirements. 
It  can  also  aid  the  operational  suitability  test 
planner  in  several  ways.  For  a  system  that 
is  under  the  reporting  provisions  of  AFR 
65-110  during  test,  the  reporting  system  be¬ 
comes  a  detailed  historical  record  of  avail¬ 
ability  and  utilization  rates.  For  reporting 
purposes,  once  the  initial  inventory  files  have 
been  created,  reports  are  submitted  only 
when  changes  in  inventory  or  status  occur 
and/or  when  the  equipment  is  used.  Inven¬ 
tory,  status,  and  utilization  reports  are 
rendered  daily,  covering  the  preceding  24- 
hour  period.  The  maintenance  activity 
reports  inventory  and  status  while  operations 
report  utilization.  Exceptions  to  these  proce¬ 
dures  apply  to  communication-electronic- 
meteorological  (CEM)  equipment,  which  uses 
AF  Form  264  (MMICS  Job/Status  Document) 
to  report  inventory  and  status,  and  to  the  on¬ 
line  inventory  and  reporting  procedures 
under  MMICS.  The  system  can  be  used  for 
subsystems  when  special  arrangements  have 
been  made. 

(5)  Standard  Base  Supply  Svstem 
(SBSS): 

(a)  The  SBSS  is  an  automated  inven¬ 
tory  accounting  system,  designed  to  provide 
timely  support  to  base-level  activities.  The 
SBSS  uses  a  computer  for  storage  and  main¬ 
tenance  of  records  and  for  generation  of 
management  reports.  AFM  67-1  provides  an 
overview  of  the  SBSS. 

(b)  SBSS  has  the  capability  to  process 
demands  and  receipts,  compute  levels  of 
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supply,  and  control  inventories.  The  system 
produces  numerous  reports  on  a  scheduled 
and  an  as-required  basis  which  are  designed 
to  assist  managers  at  all  levels  in  the  dis¬ 
charge  of  their  responsibilities.  Of  course, 
SBSS  data  are  only  useful  when  supply 
support  is  organic.  It  is  frequently  not 
useful  for  IOT&E,  when  other  supply  meas¬ 
urement  system  must  be  developed. 

(6)  Common  Data  Extraction  Programs 
(CDEP): 

(a)  The  logistics  composite  model 
(LCOM)  requires  detailed  information  con¬ 
cerning  the  frequency  of  maintenance  tasks 
and  resource  requirements.  The  frequency 
and  resource  requirements  of  scheduled  main¬ 
tenance  tasks  can  normally  be  obtained  from 
applicable  technical  orders.  However,  the 
frequency  of  unscheduled  maintenance  tasks 
and  resource  requirements  are  not  readily 
available  from  any  source.  To  solve  this 
problem,  TAC  and  the  Human  Resource  Lab¬ 
oratory  (HRL)  developed  computer  programs 
which  process  CAMS  (see  AFM  66-279,  Core 
Automated  Maintenance  System  (CAMS)) 
base-level  history  tapes  (ABD6DA).  The 
output  from  the  TAC  program  was  frequency 
information  for  unscheduled  maintenance 
tasks.  The  output  from  the  HRL  program 
was  frequency  of  unscheduled  maintenance 
tasks  and  resource  information. 

(b)  To  standardize  LCOM  methodology, 
the  Air  Force  Maintenance  and  Supply  Man¬ 
agement  Team  (AFMSMT)  product  d  a  stand¬ 
ard  version  of  the  data  extraction  programs. 
The  logic  from  the  HRL  programs  was  used 
as  the  basis  for  CDEP.  However,  several 
enhancements  were  added  to  improve  the 
utility  of  CDEP  for  a  variety  of  applications. 

(c)  The  base-level  history  tape 
(ABD6DA)  contains  maintenance  data  collec¬ 
tion  (MDC)  data.  These  data  are  taken 
directly  from  AFTO  Forms  349,  Maintenance 
Data  Collection  Record,  prepared  by  the 
maintenance  technicians.  As  a  result,  the 
base-level  tapes  represent  MDC  data  in  the 
most  elemental  form.  However,  there  is  a 
large  gap  between  the  data  requirements  of 
LCOM  and  data  contained  on  the  base-level 
tape.  Basically,  this  gap  is  created  by  the 
very  detailed  reporting  procedures  of  the 
MDCS.  The  main  function  of  the  CDEP  is 
to  analyze  MDC  records  and  present  them  in 
a  form  compatible  with  LCOM  networking 
methodology.  Detailed  instructions  for  use  of 
CDEP  are  documented  in  the  common  data 
extraction  program  standard  system  user 
documentation  report  which  AFMSMT, 
Wright-Patterson  AFB,  Ohio,  writes  and 
distributes. 

(7)  OMNIVORE/MICRO-OMNIVORE: 


ia)  OMNTVORE/MICRO-OMNTVORE  is 
an  AFOTEC-developed  data  retrieval  and 
analysis  system.  The  system  enables  the 
user  to  consolidate  selected  maintenance  and 
operating  time  data  collected  in  standard  Air 
Force  automated  data  systems  and  to  per¬ 
form  detailed  statistical  analyses  which  are 
not  available  from  existing  standard  systems. 

(b)  The  system  is  currently  imple¬ 
mented  to  accept  data  inputs  from  CAMS. 
MMICS,  and  SEDS.  Proper  use  of 
OMNIVORE/MICRO-OMNIVORE  requires 
some  knowledge  and  familiarity  by  the  func¬ 
tional  user  with  the  purpose  and  capabilities 
of  these  four  systems. 

(c)  Reference  AFOTEC  operating  in¬ 
structions  for  additional  guidance. 

(8)  Limitations  of  AFLC  Products.  Prod¬ 
ucts  from  AFLC  data  systems  can  be  of 
significant  benefit  for  logistics  assessments 
during  test  planning  and  conduct.  However, 
the  lack  of  timeliness  limits  their  usefulness 
in  preparing  evaluation  reports  at  the  end  of 
test.  The  AFLC  reports  are  published  ap¬ 
proximately  60  days  after  the  closeout  date 
for  each  monthly  or  quarterly  cycle.  With 
final  test  reports  due  within  60  days  after 
completion  of  test  program  (IOT&E/FOT&E), 
data  from  the  last  2  months  of  operational 
testing  would  not  be  available  for  inclusion 
in  the  test  report.  The  impacts  of  this  data 
shortage  on  test  reporting  should  be  evalu¬ 
ated  to  determine  usefulness  of  these  prod¬ 
ucts  during  OT&E. 

b.  AFOTEC  Point  of  Contact.  The 
AFOTEC  point  of  contact  concerning  data 
collection  and  processing  is  the  Logistics 
Studies  and  Analysis  Division  (HQ  AFOTEC/ 
LG4).  Questions  regarding  selection  of  the 
appropriate  data  collection  system  and  imple¬ 
mentation  during  test  should  be  directed  to 
this  organization. 

7-4.  Other  Data  Sources.  During  multi- 
service  test,  AFOTEC  personnel  may  use  US 
Army  or  Navy  data  systems,  particularly  if 
the  Air  Force  is  not  the  lead  service.  The 
following  discussion  identifies  Army  and 
Navy  (including  Marine  Corps)  systems: 

a.  US  Army  Data  Systems.  The  two 
most  widely  used  data  base  management 
systems  within  the  US  Army  are  System 
2000  and  MANAGE.  In  addition,  the  US 
Army  uses  reliability,  availability,  and  main¬ 
tainability  engineering  data  bases  to  store 
and  retrieve  data.  These  data  bases  include 
reliability,  availability,  maintainability,  and 
logistics  (RAM/LOG);  vehicle  technical  man¬ 
agement  system  (VETMIS);  tank-automotive 
integrated  data  base  (TAIDB);  common  test 
data  collection  system  (CTDCS);  and 
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MANAGE. 

b.  Navy  Data  Systems: 

(1)  Naval  Aviation  Maintenance  and 
Material  Management  System  (Aviation  3-M). 
The  Naval  aviation  maintenance  and  materi¬ 
al  management  system  is  a  comprehensive 
system  which  combines  availability,  reliabil¬ 
ity,  maintainability,  and  logistics  support- 
ability  into  a  single  data  system.  The  3-M 
system  uses  labor  hours  from  the  man-hour 
accounting  (MHA)  system,  maintenance 
activity  from  the  maintenance  data  reporting 
(MDR)  system,  mission-capable  status  from 
the  subsystem  capability  impact  reporting 
(SCIR)  system,  ground  support  equipment 
inventory  and  utilization  from  the  ground 
support  equipment  (GSE)  reporting  system, 
supply  support  from  the  material  reporting 
(MR)  system,  and  training  device  utilization 
and  support  from  the  training  device  support 
data  (TDSSD)  system.  The  bulk  of  the  input 
data  is  recorded  on  visual  information  display 
system/maintenance  action  form  (VTDS/MAF) 
which  contains  maintenance  and  supply  data. 

(2)  Navy  Ships’  Maintenance  and  Mate¬ 
rial  Management  System.  The  Navy  ships’ 
3-M  system  is  analogous  to  the  aviation  3-M 
system.  It  contains  maintenance  data  collec¬ 
tion  systems  for  both  active  and  mothballed 
equipment,  a  maintenance  data  processing 
system,  and  an  alteration  management 
system  for  shipboard  configuration  control. 
AFOTEC  OT&E  may  encounter  the  surface 
ship  system  in  evaluating  such  systems  as 
CEM  equipment  or  cruise  missiles.  The 
shipboard  3-M  system  compiles  maintenance 
actions,  labor  hours,  and  equipment  status  to 
produce  reports  of  work  center  production, 
system  and  component  maintenance  sum¬ 
maries,  and  other  maintenance  management 
aids. 

c.  Government-Industry  Data  Ex¬ 
change  Program  (GIDEP).  The  GIDEP 
is  a  cooperative  activity  between  government 
and  industry  participants  to  automatically 
exchange  certain  types  of  technical  data 
essential  in  the  research,  development,  pro¬ 
duction,  and  operational  life  cycle  of  systems 
and  equipment.  The  GIDEP  maintains 
specialized  data  banks  which  are  available  to 
government  and  industry. 

d.  Defense  Logistics  Studies  Infor¬ 
mation  Exchange  (DLSIE): 

(1)  DODI  5154.19,  Defense  Logistics 
Studies  Information  Exchange  (DLSIE), 
formed  the  basis  for  its  current  charter.  The 
Air  Force  implementing  regulation  is  AFR 
400-51,  Operation  of  the  Logistics  Research 
Program. 

(2)  DLSIE  collects,  organizes,  stores,  and 
disseminates  information  pertaining  to  logis¬ 


tics  studies,  models,  management  informa¬ 
tion,  and  related  documentation  which  may 
be  of  benefit  to  the  DOD  logistics  manage¬ 
ment  and  research  community.  By  reviewing 
the  DLSIE  collection,  logistics  research  activ¬ 
ities  can  avoid  spending  defense  dollars  on 
research  that  has  already  been  done. 

e.  Defense  Technical  Information 
Center  (DTIC): 

(1)  DTIC,  a  primary  field  activity  of  the 
Defense  Supply  Agency,  is  the  central  bank¬ 
ing  institution  for  DOD’s  collection  of  re¬ 
search  and  development  information  in 
virtually  all  fields  of  science  and  technology. 

(2)  DTIC  has  the  mission  to  exploit  the 
contents  of  its  collection  to  answer  three 
basic  questions  relative  to  the  research, 
development,  test  and  evaluation  (RDT&E) 
program  of  DOD: 

(a)  What  research  is  being  planned? 

(b)  What  research  is  currently  being 
performed? 

(c)  What  results  were  realized  by 
completed  research? 

(3)  The  AFOTEC  historian  (HQ  AFOTEC/ 
RS  (Research  Services))  is  the  focal  point  for 
dealing  with  DTIC.  HQ  AFOTEC/RS  main¬ 
tains  a  significant  number  of  DTIC  docu¬ 
ments  and  periodically  receives  the  catalog 
of  new  publications  from  DTIC. 

f.  Military  Specification,  Handbooks, 
and  Standards: 

(1)  A  logistics  evaluator  will  frequently 
need  access  to  military  specifications  (MIL- 
SPEC),  handbooks  (MIL-HDBK),  and  stand¬ 
ards  (MIL-STD).  These  documents  are  refer¬ 
enced  in  contracts,  requests  for  proposals 
(RFP),  system  specifications,  and  integrated 
logistics  support  plans  (ILSP). 

(2)  HQ  AFOTEC/LG  has  established  a 
technical  library  which  contains  some  of  the 
moot  commonly  used  documents.  If  you  need 
access  to  documents  not  contained  in  this 
library,  contact  HQ  AFOTEC/RS. 

g.  AFOTEC  OT&E  Data  Bank.  HQ 
AFOTEC/RS  manages  the  AFOTEC  OT&E 
data  bank.  The  data  bank  is  a  research  and 
reference  service  supported  by  microfiche 
documents,  technical  paper,  studies,  analyses, 
test  plans,  evaluations,  policy  papers,  and 
specific  program  documents.  Several  hundred 
hard-copy  reports  from  a  variety  of  sources 
are  also  on  hand. 

7-5.  Logistics  Support  Analysis  Pro¬ 
gram: 

a.  Analytical  Efforts.  The  analytical 
efforts  which  support  the  integrated  logistics 
support  (ILS)  process  are  referred  to  as 
logistics  support  analyses  (LSA).  A  data 
system  called  an  LSA  record  (LSAR)  captures 
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the  data  generated  during  LSA. 

b.  Planning  and  Execution.  LSA  is 
important  to  suitability  test  planning  and 
execution  because  it  is  the  major  source  of 
data  used  to  develop  the  support  system.  A 
series  of  15  LSA  tasks  and  77  subtasks 
detailed  in  MIL-STD  1388- 1A,  Logistics  Sup¬ 
port  Analysis,  can  be  selectively  chosen  for 
the  LSA  effort  based  on  unique  program 
requirements.  The  selected  tasks  will  pro¬ 
duce  detailed  maintenance  task  information 
which  is  useful  in  the  OT&E  logistics  analy¬ 
sis  effort.  It  also  shows  the  detailed  factors 
on  which  the  support  system  is  based.  These 
can  be  used  to  reveal  potential  problem 
areas.  LSA  inputs  and  outputs  are  sum¬ 
marized  in  figure  7-1. 

c.  Primary  Objectives: 

(1)  The  analysis  identifies  the  qualita¬ 
tive  and  quantitative  logistics  support  re¬ 
quirements.  A  systematic,  comprehensive 
analysis  is  conducted  on  an  interactive  basis 
throughout  the  system’s  life  cycle.  Initial 
analyses  evaluate  the  system’s  design  and 
operational  parameters  and  estimate  support 
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o  Operational  and  mainte-  o 

nance  requirement 


o  Item  reliability  and  o 

maintenance  charac¬ 
teristics 

o  Task  analysis  summary  o 

o  Maintenance  and  opera-  o 

tor  analysis 

o  Support  and  test  equip-  o 

met  or  training  de¬ 
scription  and  justifi¬ 
cation 

o  Facility  description  o 

justification 

o  Skill  evaluation  o 

justification 

o  Supply  support  require-  o 

ments 


costs.  During  the  development  phase,  main¬ 
tenance  tasks  are  defined  and  the  logistics 
support  requirements  are  identified.  During 
the  operational  phase,  proposed  changes  and 
modifications  are  evaluated  to  identify  their 
effect  on  maintenance  and  support. 

(2)  The  analysis  influences  the  system’s 
design  for  logistic  considerations.  The  initial 
analysis  effort  evaluates  the  effects  of  design 
alternatives  on  support  costs  and  operational 
readiness.  Known  scarcities,  constraints,  or 
logistics  risk  are  identified,  and  ways  of 
overcoming  or  minimizing  them  are  devel¬ 
oped.  During  FSD,  the  analysis  is  oriented 
toward  assisting  the  designer  in  improving 
supportability,  ease  of  maintenance,  and 
ensuring  the  logistics  infrastructure  is  devel¬ 
oped  and  in  place  to  support  the  fielding  of 
the  system. 

(3)  The  analysis  communicates  require¬ 
ments  and  integrates  the  elements  of  logistics 
support  into  a  logistic  support  system.  The 
LSA  program  establishes  a  communication 
link  between  the  hardware  design  and  ILS 
functional  organizations  through  the  LSAR. 


OUTPUTS 

Direct  annual  maintenance  and 
operator  man-hours  by  skill  and 
specialty  and  level  of  mainte¬ 
nance 

Reliability  and  maintainability 
summaries  and  analyses 


Facility  requirements  summary 

Support  equipment  requirement 
maintenance  level/WUC 

Special  tools  list 

Requirements  for  special  training 
devices 


Requirements  for  facilities 


Personnel  and  skill  summary 
Training  task  list 

Provisioning  data 


Figure  7-1.  LSA  Inputs  and  Outputs 
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LSA  is  a  source  of  data  for  the  system 
design  effort  in  the  form  of  suggestions  for 
improving  the  reliability,  maintainability, 
logistics  supportability,  and  ease  of  main¬ 
tenance.  The  LSAR  provides  data  for  risk 
analyses,  effectiveness  studies,  design  and 
logistics  support  tradeoffs,  and  life  cycle  cost 
analyses. 

d.  Process.  The  contractor  performs  the 
LSA  by  applying  the  guidelines  of  MIL-STD 
1388.  The  LSA  process  is  iterative,  begin¬ 
ning  in  the  concept  exploration  phase  with 
identification  of  support  constraints  for  the 
system.  The  emphasis  shifts  from  one  anal¬ 
ysis  task  to  another  as  design  progresses  and 
data  requirements  increase.  For  example,  in 
the  early  development  stages,  when  design 
requirements  are  not  well-defined,  the  analy¬ 
sis  is  directed  toward  identifying  and  estab¬ 
lishing  parameters  for  support  functions  at 
a  system/subsystem  level.  After  hardware 
configurations  are  defined,  the  analysis  effort 
becomes  more  comprehensive  and  is  directed 
toward  the  line-replaceable  unit  (LRU)  or 
piecepart  level.  Well  before  deployment,  the 
analysis  concentrates  on  the  impact  on  the 
supply  system  and  maintenance  organizations 
and  develops  detailed  support  requirements. 

e.  LSAR: 

(1)  The  LSAR  is  a  medium  for  system¬ 
atically  recording  analysis  data.  The  LSAR 
may  be  used  on  any  program,  regardless  of 
size  or  complexity.  The  LSAR  may  be  auto¬ 
mated  or  nonautomated.  The  formats  and 
data  element  definitions  for  automated  LSAR 
as  specified  in  MIL-STD  1388-2A,  Logistics 
Support  Analysis  Record,  DOD  requirements 
form  may  be  amended,  supplemented,  or 
altered  with  procuring  activity  approval  to 
tailor  them  to  program  variations.  The 
procuring  activity  must  specify  which  data 
elements  are  required  for  the  particular 
application. 

(2)  Data  provided  by  the  procuring  activ¬ 
ity,  generated  by  coincident  engineering  re¬ 
quirements,  and  derived  through  the  LSA  are 
input  to  a  standardized  LSA  automatic  data 
processing  (ADP)  program  through  input  data 
sheets.  These  input  data  are  used  by  the 
ADP  program  to  generate  automated  LSARs. 
Data  sources  for  selected  blocks  on  each 
input  sheet  may  be  clearly  indicated  by  the 
use  of  a  coding  system.  These  data  sheets, 
structured  for  a  particular  acquisition  pro¬ 
gram,  will  be  filled  in  as  data  become  avail¬ 
able.  Such  data  sheets  also  act  as  checklists 
to  ensure  the  analysis  provides  adequate 
visibility  of  logistic  support  resource  require¬ 
ments  at  all  levels  of  hardware  indenture. 

(3)  LSARs,  which  are  outputs  from  data 
accumulated  in  the  LSA  data  base  and  LSA 


tasks,  may  be  of  varying  types  depending  on 
the  individual  acquisition  program.  Some 
records  will  proride  working  data  to  contrac¬ 
tor  personnel  responsible  for  some  of  the 
coincident  programs  affecting  the  LSA  e.g.. 
human  engineering;  package,  handling, 
storage,  and  transportation;  technical  data; 
and  personnel  and  training.  Other  records 
may  be  generated  in  direct  response  to  data 
item  descriptions  called  out  on  the  contractor 
data  requirements  list  ('CDRL;. 

7-6.  Combining  DT&E  and  OT&E  Data: 

a.  An  R&M  data  base  separate  from  the 
DT&E  agency  is  undesirable.  AFR  800-18 
places  the  responsibility  for  implementing 
R&M  data  collection  systems  during  DT&E 
or  OT&E  on  AFSC  and  requires  the  data 
base  be  available  to  all  agencies  partici¬ 
pating  in  the  test  program. 

b.  A  larger  data  base  usually  gives  the 
logistics  analyst  more  accuracy  and  more 
confidence  in  assessing  system  characteristics. 
However,  there  are  pitfalls  associated  with 
larger,  aggregate  data  bases.  Any  DT&E  or 
OT&E  data  point  which  is  placed  in  an 
aggregate  data  base  must  adhere  to  the 
applicable  assumptions  of  the  various  models 
being  used.  Therefore,  the  development  of  a 
meaningful  aggregate  data  base  requires 
ground  rules  be  established.  Further,  the 
developer,  user,  and  tester  must  jointly 
review  available  data  and  deride  what  data 
to  place  in  an  aggregate  data  base.  This  is 
done  by  a  joint  reliability  and  maintainability 
evaluation  team  (JRMET).  In  multiserrice/ 
multiagency  DT&E,  where  the  Air  Force  is 
not  the  lead  agency,  a  JRMET  equivalent 
group  should  be  used  (i.e.,  the  JRMET  may 
be  renamed  as  long  as  the  basic  purpose 
remains  intact). 

7-7.  Joint  Reliability  and  Maintain¬ 
ability  Evaluation  Team: 

a.  Air  Force  policy  dictates  test  and  opera¬ 
tional  data  collection  and  analysis  systems  be 
complementary  to  each  other  to  verify  R&M 
performance  throughout  the  system  or  equip¬ 
ment  life  cycle.  The  purpose  of  the  JRMET 
is  to  review  raw  R&M  data  for  accuracy, 
completeness,  and  contractual  and  operational 
relevance  and  to  ensure  data  collection  and 
analysis  systems  are  complementary. 

b.  The  implementing  command  is  respon¬ 
sible  for  establishing  a  JRMET,  writing  the 
JRMET  charter,  and  convening  JRMET 
meetings.  The  system  program  office  (SPO) 
chairs  the  JRMET,  which  is  composed  of 
representatives  of  the  SPO,  supporting  com¬ 
mand,  operating  command,  DT&E  test  team, 
OT&E  test  team,  other  participating  service/ 
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agency  representatives,  and  when  appro¬ 
priate,  contractor  personnel.  The  latter  when 
present  serves  only  in  an  advisory  capacity. 

c.  Specific  responsibilities  of  JRMET 
participants  are  specified  in  the  JRMET 
charter.  The  charter  also  states  policy  for 
the  joint  use  of  R&M  data,  exchanges  of 
R&M  information,  classification  criteria  for 
system  related  failures/faults,  and  adminis¬ 
trative  procedures  for  conducting  JRMET 
meetings.  The  intent  of  having  a  charter  is 
to  avoid  duplication  of  effort.  If  a  charter  is 
not  established,  the  JRMET  functions  nor¬ 
mally  will  be  assigned  within  the  SPO. 
AFOTE C/LG4  has  several  strawman  JRMET 
charters  for  use  when  interfacing  with  the 
SPO. 

d.  The  importance  of  a  JRMET  in  OT&E 
cannot  be  over  emphasized.  JRMET  meet¬ 
ings  serve  as  an  ideal  forum  for  reviewing 
R&M  data,  becoming  familiar  with  R&M 
terminology  (see  part  three),  obtaining  com¬ 
mon  agreement  on  the  use  of  R&M  data,  and 
ensuring  data  accuracy.  JRMET  meetings 
should  be  held  periodically  as  R&M  data  are 
collected.  Actual  operational  suitability  T&E 
data  reduction,  analysis,  and  evaluation  are 
normally  not  done  at  the  meetings;  rather, 
the  DLE  and  logistics  TSG  personnel  should 
accomplish  this  after  each  JRMET  meeting 
to  obtain  estimates  of  applicable  suitability 
MOEs.  Release  of  any  preliminary  OT&E 
data  analysis  to  agencies  outside  AFOTEC 
must  be  coordinated  with  the  test  director, 
test  manager,  and  AFOTE  C/PA  as  specified 
in  the  test  plan. 

e.  Figure  7-2  is  an  example  of  a  typical 
JRMET  organization.  AFOTEC  involvement 
in  the  JRMET  entails  understanding  the 
data  for  OT&E  purposes  and  assisting  AFSC 
and  the  contractor  in  understanding  the  data 
from  an  operational  perspective.  While 
AFOTEC  is  not  bound  to  contractual  inter¬ 
pretation  of  test  data,  agreements  are  usually 
reached  to  establish  a  common  data  base  for 
use  by  all  participating  agencies.  As  test 
team  DLE/DSEs  may  be  unfamiliar  with  the 
JRMET  organization  and  functioning,  they 
should  review  the  JRMET  charter  and  con¬ 
vene  a  pre- JRMET  consisting  of  test  team 
logistics  personnel,  logistics  TSG  personnel, 
and  appropriate  operating  command  person¬ 
nel.  Air  Force  OT&E  position  with  respect 
to  the  classification  of  R&M  test  data:  The 
pre-JRMET  is  normally  held  one  day  before 
the  formal  JRMET  meeting.  Key  questions 
to  answer  at  the  pre- JRMET/ JRMET  meet¬ 
ings  are:  Is  a  failure  relevant  or  test  pecu¬ 
liar?  Did  the  failure  critically  impact  the 
mission?  Do  computed  values  of  R&M  terms 
relate  to  contractual  or  operational  require¬ 


ments?  Are  values  of  R&M  terms  expressed 
as  progress  points  on  a  growth  curve  or  as 
mature  system  values?  Was  the  failure  soft¬ 
ware  related?  Has  a  failure-related  service 
report  been  written,  or  is  one  needed? 

f.  In  summary,  the  JRMET  allows 
AFOTEC  to  reconcile  data  differences  with 
the  SPO  or  contractor.  Other  agencies  also 
benefit  by  gaining  the  latest  test  information 
to  use  in  updating  logistics  plans,  logistics 
support  analysis,  initial  provisioning  esti¬ 
mates,  and  life  cycle  cost  estimates.  In  addi¬ 
tion,  the  integrated  logistics  support  manage¬ 
ment  team  (ILSMT)  or  resident  integrated 
logistics  support  activity  (RILSA)  interfaces 
with  the  JRMET  to  give  the  Air  Logistics 
Center  (ALC)  updated  system  program  man¬ 
agement  information. 

7-8.  Lessons  Learned: 

a.  The  DMAP  should  parallel  development 
of  the  test  plan  and  be  completed  before  the 
start  of  active  testing  and  should  specify,  for 
each  MOE,  the  required  data  elements,  data 
processing  procedures,  and  analysis  methodol¬ 
ogy.  A  "DMAP  dry  run"  should  be  accom¬ 
plished  by  the  test  team  and  TSG  prior  to 
the  test  readiness  briefing.  An  inadequate 
DMAP  will  allow  incomplete  or  inaccurate 
data  to  be  entered  into  the  data  system;  data 
collection  procedures  that  are  not  compatible 
with  the  data  system;  or  unacceptable  delays 
in  data  processing,  analysis,  or  reporting. 

b.  Early  DMAP  items  requiring  TBD 
action  by  the  test  team  should  be  so  identi¬ 
fied.  The  DMA P  must  be  flexible  enough  to 
accommodate  lessons  learned  in  test  startup 
to  changes  to  the  acquisition  program  itself 
as  well  as  corresponding  changes  to  the 
objectives.  Similarly,  DMAP  procedures  have 
frequently  required  test  team  adjustment  as 
they  determine  the  effect  test  support  or  the 
environment  has  on  the  test  conduct.  Exer¬ 
cising  the  system  with  "dry  runs"  can  un¬ 
cover  problems  early  and  enable  the  test 
team  to  revise  the  DMAP  or  refine  analysis 
methods  or  collection  procedure  while  test 
time  and  resources  are  still  available.  There¬ 
fore,  DMAP  development  should  consider  an 
iterative,  cooperative,  and  continuous  process 
among  the  test  team,  test  managers,  mission 
planners,  data  managers,  and  resources 
requirements  planners. 

c.  Several  tests  have  shown  collected  data 
may  not  always  be  as  expected.  This  will  be 
particularly  true  when  OT&E  is  combined 
with  DT&E  and  the  evaluation  is  based,  at 
least  in  pent,  on  contractor  data.  Test  data 
requirements  should  be  coordinated  as  early 
as  possible  with  the  SPO,  who  has  the 
authority  to  impose  requirements  on  the 
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contractor.  This  is  also  true  when  data  are 
obtained  from  any  other  agency.  Trial  runs 
of  the  data  should  uncover  any  data  deficien¬ 
cies  early  enough  to  allow  their  correction. 

d.  The  objective  of  the  test  team  is  to 
evaluate  a  system,  not  just  operate  it.  All 
test  team  members  should  appreciate  that 
data  collection  and  processing  is  an  essential 
part  of  this  mission.  Data  collectors  should 
be  involved  in  the  test  planning  and  should 
be  permanently  assigned  to  the  test  team  as 
data  collectors. 

e.  The  DMAP  should  address  require¬ 
ments  for  storage  of  classified  material, 
particularly  for  a  test  that  will  be  conducted 
in  several  locations. 

f.  Combined  DT&E/OT&E  should  have  a 
single  data  manager,  normally  within  the 
DT&E  organization.  OT&E  should  also  have 
a  data  focal  point. 

g.  The  DMAP  should  emphasize  collect¬ 
ing  data  while  they  are  still  fresh  in  the 
tester’s  mind.  The  tester  should  conduct  a 
prompt  quality  check  to  ensure  accuracy  and 
completeness. 

h.  To  be  effective,  LSA  has  to  be  mutually 
understood  by  the  SPO  and  the  contractor. 
If  this  understanding  does  not  exist,  the 
contractor  may  just  amass  data  for  govern¬ 
ment  use  without  using  the  data  to  influence 
system  design  or  the  development  of  the 
logistics  support  elements.  Early  and  en¬ 
lightened  involvement  of  AFOTEC  logistics 
personnel,  in  addition  to  SPO  logistics  per¬ 
sonnel,  can  help  avoid  this  problem. 

i.  Audits  of  LSA  data  and  methods  are 
important.  AFOTEC  is  often  in  a  position  to 
help  the  SPO  audit  the  contractor’s  LSA 
Correcting  LSA  deficiencies  (e.g.,  optimistic 
failure  rates  and  faulty  assumptions)  early 
enables  a  more  realistic  support  posture  to 
be  developed. 

j.  LSA  is  supposed  to  be  an  iterative  proc¬ 
ess  with  early  projections  of  R&M  parameters 
updated  with  demonstrated  values  as  the 
system  develops.  Some  contractors  are 
reluctant  to  make  these  updates.  When  this 


happens,  the  SPO  should  be  convinced  to 
require  the  contractor  to  update  the  data 
base  with  test  data  as  they  become  available. 

k.  When  LSA  data  are  to  be  the  primary 
source  of  data  for  OT&E  analysis  (e.g.. 
LCOM),  review  this  data  base  early  to  iden¬ 
tify  deficiencies  to  ensure  the  data  are  us¬ 
able.  Potential  areas  for  review  include: 

(1)  Level  of  aggregation  too  high  and 
detail  lacking. 

(2)  Full-scale  development  (FSD)  and 
production  data  mixed  without  distinction. 

(3)  Maintenance  activities  do  not  track 
through  the  data  base  (e.g.,  removals  with  no 
shop  actions). 

(4)  Inconsistent  data  (e.g.,  MTBM  greater 
than  MTBF,  system  more  reliable  than 
subsystems,  and  subsystems  more  reliable 
than  components). 

(5)  LSA  reports  difficult  to  read  and 
reports  poorly  labeled. 

(6)  Data  not  current. 

(7)  Wartime  scenarios  not  considered. 

l.  Understanding  the  purpose  and  use  of 
the  R&M  data  collected  is  fundamental  to 
the  proper  functioning  of  the  JRMET.  A 
clear  definition  of  failures  and  R&M  terms  is 
essential.  In  the  past,  problems  have  oc¬ 
curred  because  users  simply  misunderstood 
the  intent  of  the  definitions,  and  it  was  not 
uncommon  for  independent  evaluations  of  a 
system  to  use  different  failure  definitions, 
resulting  in  significantly  different  R&M 
values. 

m.  Scheduling  frequent  meetings  of  the 
JRMET  increases  travel  costs,  whereas  infre¬ 
quent  meetings  results  in  long,  laborious 
reviews  of  reams  of  test  data.  The  DLE  can 
aid  in  balancing  the  frequency  of  meetings  by 
notifying  the  SPO  on  the  volume  of  test  data 
collected  and  suggesting  when  a  meeting 
should  be  called.  Also,  test  planning  should 
investigate  the  capability  of  having  on-line 
adjudication  of  R&M  data  with  JRMET 
meetings  called  only  to  resolve  adjudication 
problems. 
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Chapter  8 

DEVELOPING  OPERATIONAL  SUITABILITY  REPORTS 


8-1.  General; 

a.  Discussions  in  previous  chapters  and 
AFR  55-43,  establish  numerous  reporting 
requirements  to  ensure  proper  test  planning, 
test  execution,  and  communication  of  test 
results  to  decision  makers.  The  TSG  and 
test  team  need  to  be  aware  of  these  reporting 
requirements,  the  associated  schedules,  and 
report  formats.  In  case  of  briefings,  addi¬ 
tional  considerations  apply,  i.e.,  reserving  a 
conference  room,  obtaining  slide  projector  or 
viewgraph  equipment,  etc.  HQ  01  11-6 
establishes  procedures  for  presenting  brief¬ 
ings  to  the  command  section. 

b.  This  chapter  is  not  meant  to  outline 
briefing  procedures  such  as  know  your  audi¬ 
ence,  rehearse  your  presentation  with  peers, 
review  your  slides  for  typos  and  proper  flow 
of  information,  etc.  It  is  also  not  meant  to 
teach  you  how  or  what  to  write;  rather  it 
explains  the  key  test  reports,  gives  considera¬ 
tions  for  presenting  the  operational  suit¬ 
ability  information,  and  provides  some  les¬ 
sons  learned. 

c.  A  successful  report  requires  an  audit 
trail  back  through  data  analysis  and  reduc¬ 
tion,  raw  test  data,  test  planning  assump¬ 
tions,  and  limitations  to  the  operational 
requirement.  It  also  requires  communication 
between  the  test  team,  the  TSG,  the  SPO, 
and  the  supporting  and  operating  commands. 

8-2.  Operational  Suitability  Reports: 

a.  There  are  four  key  reports: 

(1)  Service  Reports.  This  report  docu¬ 
ments  system  deficiencies  and/or  enhance¬ 
ments  discovered  during  OT&E.  Procedures 
are  explained  in  TO  00-35D-54  and  AFR 
55-43.  The  test  team  should  submit  service 
reports  for  both  operational  effectiveness  and 
operational  suitability  deficiencies/enhance¬ 
ments. 

(2)  Activity  Reports.  These  reports  are 
messages  between  the  test  team  and  the 
TSG  to  communicate  key  test  events  com¬ 
pleted/planned,  problems  associated  with  the 
OT&E,  projected  test  completion  date, 
changes  in  key  test  personnel,  and  OT&E 
specific  information. 

(3)  Interim  Reports.  These  reports  pro¬ 
vide  current  test  results,  test  conclusion  and 
evaluation  information  at  designated  points 
during  test  execution  (midway  through  the 
test,  at  the  end  of  a  test  phase,  etc.).  A 
message  format  is  normally  used.  The 
Software  Analysis  Division  software  evalua¬ 


tion  managers  (SEM),  deputies  for  software 
evaluation,  and  software  evaluation  team 
members  (SETM)  conduct  AFOTECP  800-2 
series  evaluations  throughout  the  software 
development  life  cycle.  The  results  of  these 
initial  assessments  are  written  in  software 
interim  reports.  These  interim  reports  are 
staffed  and  provided  to  TE  for  distribution 
beyond  AFOTEC.  They  are  also  used  in 
support  of  the  IOT&E  final  report. 

(4)  Final  Reports.  The  final  report  pre¬ 
sents  results,  conclusions,  and  recom¬ 
mendations  from  OT&E.  A  report/interim 
summary  briefing,  summarizing  test  results, 
may  be  required  to  support  program  decision 
milestones  when  insufficient  time  exists  to 
prepare  the  final  report.  A  separate  data 
document  is  used  to  provide  detailed  test 
information. 

b.  Briefings  are  required  for  test  planning 
reviews  (TPR),  final  OT&E  plan,  test  readi¬ 
ness  reporting,  test  status  reporting,  opera¬ 
tional  assessments,  interim  summary  re¬ 
porting,  and  final  reporting.  The  level  of 
detail  of  these  briefings  will  vary  with  the 
audience.  In  all  cases,  briefings  will  be 
coordinated  with  HQ  AFOTEC  staff.  Addi¬ 
tionally,  the  logistics  and  software  evaluation 
managers  (and/or  the  DLE/DSE)  with  support 
from  the  logistics  analysis  manager  should  be 
prepared  to  brief  the  AFOTEC  Director  of 
Logistics  during  interim/final  report  coordina¬ 
tion  cycle. 

c.  AFR  55-43  suggests  the  use  of  data 
documents  to  support  the  final  OT&E  report. 
AFR  55-43,  AFOTEC  Supplement  1  requires 
all  important  data  (not  included  in  the  final 
report)  be  published  in  supporting  data 
documents  (SORD).  HQ  AFOTEC/LG  staff 
has  need  of  such  detailed  information/ 
analysis  for  follow-on  test  planning,  global 
studies,  and  various  other  reasons.  Past  ex¬ 
perience  has  revealed  that  the  information 
is  often  destroyed,  lost,  or  not  centralized. 
An  SDD  will  be  compiled  for  all  logistics 
areas  regardless  of  whether  a  document  is 
created  for  other  areas  of  test. 

(1)  The  SDD  provides  nonjudgmental 
comments  for  in-depth  documentation  of  test 
activities  and  supports  the  conclusions  and 
recommendations  contained  in  the  final 
report.  It  is  an  objective-by-objective  presen¬ 
tation  of  the  detailed  methodology  used  for 
test  conduct  and  data  analysis,  The  informa¬ 
tion  should  be  sufficiently  detailed  to  recreate 
the  final  report,  if  necessary. 
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(2)  The  format  for  the  SDD  should  paral¬ 
lel  that  used  for  the  final  report.  Areas  ad¬ 
dressed  should  likewise  be  addressed  in  the 
SDD.  If  there  are  no  additional  data  avail¬ 
able  or  required  for  input  to  the  SDD,  it 
should  be  so  stated  in  the  major  heading 
paragraph. 

(3)  Logistic  analysis  inputs  will  present 
a  detailed  analysis  approach  to  encompass 
reliability,  maintainability,  and  availability. 
It  will  typically  include  reliability  growth, 
mathematical  models,  or  simulation  models. 
The  reliability  growth  discussion  should 
include  model  justification  and  rationale  for 
reliability  growth  and  projections  in  general. 
The  simulation  model  discussion  should 
include  model  inputs  and  all  model  validation 
documentation.  Also  included  in  all  discus¬ 
sions  should  be  data  collection  methods, 
elements  used,  and  where  appropriate,  statis¬ 
tical  significance  or  risk. 

(4)  The  nature  of  the  logistic  supportabil- 
ity  evaluation  precludes  a  definitive  approach 
to  determining  items  for  inclusion  in  an 
SDD.  Any  or  all  of  the  logistics  support 
elements  may  be  looked  at  in  the  evaluation 
process,  and  in  many  cases  "expert  opinion" 
is  the  primary  method  of  evaluation.  Areas 
observed  to  make  the  evaluation  should  be 
addressed  with  supporting  data  if  the  sce¬ 
nario  is  not  completely  described  in  the  final 
report. 

(a)  The  prevalent  kind  of  supporting 
data  for  logistics  supportability  areas  will  be 
resultant  data  from  the  administration  of 
questionnaires  or  test  team  observations. 
Provide  examples  of  the  questionnaires  used 
with  a  summary  or  matrix  depicting  the 
compiled  data  or  responses.  In-depth  expla¬ 
nation  of  the  methodology  used  in  the  admin¬ 
istration  of  the  questionnaires  and  analysis 
of  the  data  is  appropriate.  In  any  case,  the 
SDD  is  not  meant  to  include  all  raw  data. 
Rather,  it  is  a  compilation  of  data  to  support 
the  evaluation  results. 

(b)  Other  kinds  of  support  data  include, 
but  are  not  limited  to: 

1.  Task  Evaluation  Notes.  Sum¬ 
maries  of  notes  from  log  books  or  document¬ 
ed  observations  which  were  referred  to  and 
used  for  determination  of  adequacy. 

2.  Meeting  Notes.  Action  items  from 
meetings  which  support  final  conclusions  or 
demonstrate  how  conclusions  were  reached. 
An  example  is  minutes  from  logistic  support 
meetings  which  may  demonstrate  how  infor¬ 
mation  was  obtained  to  reach  decisions. 

3.  Log  Books.  Summaries  of  entries 
into  test  log  books  which  reconstruct  the 
scenarios  and  describe  evaluation  method¬ 
ology. 


4.  Trip  Reports.  Descriptive  accounts 
of  tests  conducted  or  meetings  attended  that 
may  have  provided  information  used  in  the 
evaluation. 

5.  Photograph/Video.  Supportive 
photographic  documentation  is  acceptable. 

(c)  Integrated  diagnostic  (ID)  inputs 
will  summarize  the  data  that  were  available 
for  evaluation  and  describe,  in  detail,  how 
that  information  was  used  in  the  ID  evalua¬ 
tion.  If  data  were  not  used,  explain  why. 

1.  For  the  quantitative  area,  list  the 
formula  and  the  data  elements  that  contrib¬ 
ute  to  the  calculation  of  that  formula.  Pro¬ 
vide  an  explanation  of  how  the  data  elements 
were  categorized  in  order  to  provide  an 
understanding  of  how  the  data  were  used  to 
perform  the  ID  evaluation. 

2.  For  the  qualitative  areas  of  ID, 
include  a  copy  of  the  questionnaires  used 
and  a  summary  of  the  results  of  the  ques¬ 
tionnaires.  Any  other  information  that 
supports  the  evaluation/assessments  of  the  ID 
objectives/MOEs  should  also  be  included. 

(5)  Software  analysis  inputs  will  consist 
of  the  data,  or  appropriate  references  to 
interim  reports  containing  the  data,  that 
substantiate  the  comments  contained  in  the 
final  report. 

(a)  Use  the  test  plan  objectives  and 
measures  of  effectiveness  as  a  guide  for  data 
collection. 

(b)  As  applicable,  attach  all  interim 
reports  generated  as  a  result  of  evaluations 
conducted  in  accordance  with  the  AFOTECP 
800-2  volumes  (reference  paragraph  8-2a(3)). 

(c)  Summarize  the  data  collection 
methodology  and  results  used  to  support  the 
software  maturity,  life  cycle  process,  software 
support  resources  assessments,  and  any  other 
assessments  not  covered  by  interim  reports. 

8-3.  Report  Considerations: 

a.  Test  Team.  The  test  team  DLE/DSE 
should: 

(1)  Begin  preparing  the  final  report  on 
the  first  day  of  the  test  program. 

(2)  Be  aware  of  the  criticality  of  the  final 
report  and  begin  its  preparation  in  a  method¬ 
ical  manner  to  ensure  reporting  milestones 
are  met. 

(3)  Pay  particular  attention  to  the  execu¬ 
tive  summary  (which  is  written  last),  the 
conclusions  and  recommendations,  and  spe¬ 
cific  test  objective  results.  The  executive 
summary  should  mirror  the  body  of  the 
report.  The  conclusions  should  complement 
the  recommendations  (i.e.,  if  a  test  conclusion 
is  "technical  data  to  maintain  the  peculiar 
support  equipment  was  not  available,"  do  not 
forget  to  include  the  recommendation  "AFSC 
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procure  technical  data  for  maintaining  the 
peculiar  support  equipment").  Lastly,  the 
specific  test  results  should  describe  the 
associated  impact  on  the  mission  (e.g.,  "the 
computer’s  spare  memory  lacked  sufficient 
capacity  to  support  mission  XXX.  This 
caused  loss  of  critical  messages  being  relayed 
to  the  operator  and  forced  the  operator  to 
take  evasive  action  at  great  risk  to  himself 
and  the  aircraft"). 

(4)  Continually  educate  the  test  director 
on  operational  suitability  terminology,  logis¬ 
tics  terminology,  and  methods  of  data  analy¬ 
sis.  The  test  director  will  need  this  back¬ 
ground  for  presenting  test  results  and  con¬ 
clusions  to  HQ  AFOTEC  staff. 

(5)  Maintain  contact  with  the  logistics 
evaluation  manager,  software  evaluation 
manager,  and  logistics  analysis  manager  for 
changes  in  final  report  format,  content  phi¬ 
losophy,  and  briefing  strategies. 

(6)  Be  available  for  coordinating  and 
briefing  the  final  report  at  HQ  AFOTEC. 

b.  Logistics  TSG.  Logistics  TSG  per¬ 
sonnel  should: 

(1)  Be  available  to  assist  the  test  team 
in  writing  the  final  report. 

(2)  Obtain  test  data  periodically  to  use 


in  preparing  the  SDD. 

(3)  Coordinate  the  SDD  with  the  test 
team  before  the  team  disbands. 

8-4.  Lessons  Learned: 

a.  Suitability  test  personnel  should  not 
be  concerned  with  a  pride  of  authorship  as 
reports  are  coordinated  through  HQ 
AFOTEC. 

b.  Tailor  briefings  to  the  intended  audi¬ 
ence  by  determining  the  precise  message  to 
be  conveyed  and  stringently  controlling  the 
level  of  detail  in  the  presentation. 

c.  The  test  director,  DLE,  and  DSE  must 
be  prepared  to  present  briefings  on  short 
notice. 

d.  Host  tenant  support  agreements  (AFR 
11-4)  should  specify  procedures  for  obtaining 
graphics  support  required  in  preparation  of 
viewgraphs  for  briefings. 

e.  The  level  of  detail  of  the  final  report 
must  be  consistent  with  the  perspective  of 
the  decision  maker. 

f.  Lack  of  an  SDD  will  preclude  recon¬ 
struction  of  test  results  to  support  global- 
analysis  studies.  Such  studies  are  used  to 
improve  test  strategies  and  methodologies. 
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Part  HI 

KEY  OPERATIONAL  SUITABILITY  EVALUATION  AREAS 

Chapter  9 
AVAILABILITY 


9-1.  General: 

a.  A  recurrent  theme  in  all  guidance 
documentation  concerning  operational  suita¬ 
bility  is  the  need  to  evaluate  the  system’s 
ability  to  meet  operational  readiness  require¬ 
ments.  System  readiness  objectives  and 
operational  availability  goals  are  established 
early  in  each  acquisition  program  and  become 
the  basis  for  evaluating  logistics  support. 
Test  and  evaluation  of  the  system  provides 
data  necessary  to  determine  if  mature  system 
readiness  requirements  are  achievable.  For 
OT&E  purposes,  availability  is  generally 
considered  synonymous  with  operational 
readiness  (AFR  80-14). 

b.  System  readiness  requirements  are 
characterized  by  a  committable  condition  at 
a  specified  time  (availability),  performance  of 
mission-essential  functions  without  mission 
aborting  failures  (reliability),  and  retaining 
the  system  in  or  restoring  it  to  specified 
condition  (maintainability). 

(1)  The  first  readiness  objective,  ready 
for  commitment,  is  usually  expressed  in 
terms  of  operational  availability  (Ao).  This 
chapter  discusses  the  concept  of  availability 
and  contains  definitions  of  availability,  terms 
used  to  measure  availability,  and  mathemati¬ 
cal  expressions  of  availability.  It  presents  a 
general  approach  for  evaluating  availability 
which  emphasizes  the  importance  of  graphi¬ 
cally  describing  a  system’s  status  over  a  time 
period.  This  chapter  also  discusses  the  need 
for  simulation/modeling  and  use  of  simulation 
models  as  a  primary  tool  for  evaluating  a 
system’s  long-term  availability  or  readiness 
characteristics.  Attachment  3  presents 
specific  availability  applications. 

(2)  A  common  expression  for  the  second 
objective,  reliability,  is  weapon  system  relia¬ 
bility  (WSR).  Reliability  is  discussed  in 
chapter  10. 

(3)  For  the  third  objective,  maintain¬ 
ability,  a  common  expression  is  mean  down¬ 
time  (MDT).  Maintainability  is  discussed  in 
chapter  11. 

c.  Availability  is  the  parameter  that 
translates  the  reliability,  maintainability, 
and  logistics  supportability  characteristics  of 
the  system  into  a  measure  of  interest  to  the 
user.  It  is  based  on  the  question  "Is  the 
equipment  ready  in  a  working  condition 


when  it  is  needed?"  To  be  realistically 
evaluated,  the  evaluator  must  compare  the 
availability  of  the  system  with  mission  re¬ 
quirements  contained  in  the  statement  of 
operational  need  (ORD)  or  other  require¬ 
ments  documents. 

9-2.  Availability  Requirements: 

a.  The  ORD  and  the  requirements  corre¬ 
lation  matrix  (RCM)  specify  times  (or  ranges 
of  times)  at  which  the  system  may  be  re¬ 
quired  to  change  operating  states  or  condi¬ 
tions.  For  example,  the  system  illustrated  in 
figure  9-1  is  required  to  halt  normal  peace¬ 
time  operations  and  training,  at  time  tQ  (a 
random  time)  and  assume  a  posture  of  in¬ 
creased  readiness  prior  to  commitment  at 
time  t,.  Two  operational  availability  <A0) 
measures  appropriate  to  these  times  might 
be  (1)  the  number  of  systems  at  ”ne  t. 
which  are  capable  of  assuming  an  increased 
readiness  posture  by  time  and  ( 2 )  the 
probability  that  the  required  number  of 
systems  will  actually  assume  an  increased 
readiness  posture  by  time  tj. 

b.  The  ORD/RCM  should  also  specify  the 
expected/required  length  of  time  the  system 
must  sustain  the  increased  readiness  posture 
with  the  required  number  of  committable 
systems  (the  time  from  t.  to  t,).  Thus, 
another  measure  of  A  may  be  defined  as  the 
probability  of  sustaining  in  increased  readi¬ 
ness  posture  for  a  specified  time  with  a 
specified  number  of  mission-capable  systems. 

c.  The  mission  length  (tj  to  t3)  provides 
the  time  base  for  WSR  (i.e.,  what  critical 
failure-free  operating  times  must  be  achieved 
to  ensure  that  the  system  accomplishes  its 
mission). 

d.  Many  systems  will  be  employed  more 
than  once  in  a  wartime  scenario  (e.g.,  tactical 
fighters,  bombers,  etc.).  Therefore,  it  is 
important  to  consider  the  capability  of  the 
system  to  be  regenerated  within  an  accept¬ 
able  time  (t3  to  t4). 

e.  The  operational  availability  measures 
discussed  above  are  also  heavily  dependent 
on  the  maintenance  concept,  which  influences 
the  ability  of  the  system/equipment  to  be 
retained  in  or  restored  to  a  specified  condi¬ 
tion. 

f.  The  main  point  is  there  are  many  A0 
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definitions  which  might  be  used  for  different 
systems  at  different  times.  There  is  no 
cookbook  approach  which  will  reflect  every 
system’s  performance  in  satisfying  its  stated 
readiness  objectives. 

9-3.  "Real"  Versus  "Apparent"  Availabil¬ 
ity: 

a.  Traditionally,  availability  is  defined  as 
the  ratio  of  system  uptime  to  total  time. 
This  simple  definition  is  appropriate  for  as¬ 
sessing  systems  which  are  used  continuously 
or  those  which  incur  relatively  short  dor¬ 
mancy  periods. 

b.  Recently  the  Air  Force  has  been  devel¬ 
oping  a  number  of  systems  (primarily  mis¬ 
siles  and  munitions)  that  spend  the  vast 
majority  of  their  useful  life  in  storage  or  in 
a  dormant  state.  Failures  in  storage  may 
not  be  discovered  for  long  periods  of  time 
(e.g.,  missiles  may  be  checked  for  operability 
only  every  few  years);  for  more  complex 
systems,  failures  may  have  occurred  that  are 
undetectable  until  a  firing  event  is  actually 
attempted.  These  situations  or  state — where 
an  item  is  failed  but  undetected  (conse¬ 
quently  unavailable)  for  long  periods  of 
time — gave  rise  to  the  definitional  terms  of 
"apparent"  and  "real"  availability. 

c.  Apparent  availability  is  defined  as  the 
percentage  of  total  assets  thought  to  be 
operable,  i.e.,  perceived  as  ready  for  immedi¬ 
ate  use.  Real  availability  is  defined  as  the 
percentage  of  total  assets  that  would  actual¬ 
ly  operate  as  intended  if  the  user  began  a 
mission  execution.  The  words  "total  assets" 
are  relative  to  each  definition.  Limiting  the 
assets  to  those  under  test  will  produce  a 
very  narrow  (if  not  useless)  view  of  availa¬ 
bility.  Limiting  the  assets  to  those  assigned 
to  a  particular  squadron,  wing,  base,  etc., 
can  be  used  to  give  a  focused  (mission- sce¬ 
narios  dependent)  view  of  availability.  Last¬ 
ly,  considering  the  entire  procurement, 
phased  in  over  time,  will  produce  a  global, 
force-wide  view  of  availability. 

9-4.  Definitions: 

a.  Commonly  used  availability  elements 
are: 

ALDT  =  Administrative  and  logis¬ 
tics  downtime  spent  wait¬ 
ing  for  parts,  administra¬ 
tive  processing,  mainte¬ 
nance  personnel,  or  trans¬ 
portation  per  specified 
period. 


capable  of  performing  ail 
of  its  assigned  peacetime 
and  wartime  missions. 

MC  rate  =  Mission-capable  rate.  The 
percentage  of  possessed 
hours  a  system  is  oper¬ 
able. 

MDT  =  Mean  (maintenance)  down¬ 

time. 

MTBCF  =  Mean  time  between  criti¬ 
cal  failures. 

MTBF  =  Mean  time  between  fail¬ 

ures. 


MTBM  =  Mean  time  between  main¬ 
tenance. 

MTBUMA  =  Mean  time  between  un¬ 
scheduled  maintenance 
actions  (unscheduled  is 
generally  synonymous  with 
corrective). 

MTTR  =  Mean  time  to  repair  (AFP 
57-9  defines  this  as  a  con¬ 
tractual  term). 

MTTRS  =  Mean  time  to  restore  sys¬ 
tem. 


MRT  =  Mean  repair  time  (AFP 
57-9  defines  this  as  an 
operational  term). 

NMC  rate  =  Not-mission-capable  rate. 

The  percentage  of  pos¬ 
sessed  hours  a  system  is 
not  capable  of  performing 
any  of  its  assigned  mis¬ 
sions.  NMC  is  generally 
subdivided  by  category: 
NMC  for  maintenance 
(NMCM),  NMC  for  supply 
(NMCS),  NMC  for  both 
maintenance  and  supply 
(NMCB),  and  further  sub¬ 
divided  downtime,  by  sub¬ 
system,  or  downtime,  by 
subsystem,  or  other  divi¬ 
sions  as  required. 

OT  =  Operating  time  (equipment 

in  use). 


FMC  rate  =  Full  mission-capable  rate. 

The  percentage  of  pos¬ 
sessed  time  a  system  is 


PMC  =  Partial  mission-capable 
rate.  The  percentage  of 
possessed  time  a  system  is 
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capable  of  performing  at 
least  one,  but  not  all,  of 
its  assigned  wartime  mis¬ 
sions. 

ST  =  Standby  time  (not  oper¬ 

ating  but  assumed  oper¬ 
able)  in  a  specified  period. 

TCM  a  Total  active  corrective  (un¬ 
scheduled)  maintenance 
time  per  specified  period. 

TDT  =  Total  downtime  per  speci¬ 

fied  period  =  TMT  + 
ALDT. 

TMT  =  Total  active  maintenance 

time  per  specified  period  = 
TCM  +  TPM. 

TPM  =  Total  active  preventive 

maintenance  time  per 
specified  period  (scheduled 
is  generally  synonymous 
with  preventive). 

IT  a  Total  intended  utilization 

period  or  total  time.  [Pos¬ 
sessed  time  in  AFR  65- 
110,  Aerospace  Vehicle 
and  Equipment  Inventory, 
Status  and  Utilization 
Reporting  System  (AVI- 
SURS).] 

UR  =  Uptime  ratio.  Communi¬ 

cations,  electronics,  and 
meteorological  systems  as 
the  percentage  of  pos¬ 
sessed  time  they  are 
"operational." 

b.  Terms  used  in  the  availability  eval¬ 

uation  must  be  clearly  defined.  For  ex¬ 
ample,  assessing  the  availability  of  opera¬ 
tional  aircraft  generally  assumes  a  7-day- 
per-week  total  time  (TT)  period.  If  the 

aircraft  are  not  normally  flown  or  main¬ 
tained  on  weekends  and  are  left  in  an  "up" 
status  Friday  night,  using  a  5-day-per-week 
TT  will  generate  lower  availability  results. 

c.  Other  definitions  associated  with  avail¬ 
ability  may  also  significantly  affect  the  re¬ 
sults.  For  example,  are  "before  and  after" 
operation  checks  conducted  in  conjunction 
with  preventive  maintenance  excluded  from 
downtime  because  the  equipment  is  assumed 
operable?  How  is  administrative  and  logis¬ 
tics  downtime  determined;  is  it  assumed, 
calculated,  or  observed?  What  is  the  opera¬ 


tional  status  of  a  system  during  the  warm 
standby  period?  For  repeatability  and  com¬ 
parability  of  the  results,  all  terms  and  proce¬ 
dures  to  be  used  in  the  availability  evalua¬ 
tion  must  be  defined  in  the  test  plan. 

9-5.  Mathematical  Expressions  of  Avail¬ 
ability.  The  following  expressions  are  those 
of  the  classical  theoretical  approach.  HQ 
AFOTEC  and  test  teams  presently  use  some 
of  these.  The  basic  mathematical  definition 
of  availability  is: 

Availability  =  A  =  -UP<£S£-  = - .  uPtime - 

total  time  uptime  +  downtime 

Availability  is  assessed  by  substituting  the 
time-based  elements  defined  previously  into 
various  forms  of  the  basic  equation.  Dif¬ 
ferent  combinations  of  elements  combine  to 
formulate  different  definitions  of  availability. 
The  definitions  of  what  is  included  in  uptime 
and  what  is  included  in  downtime  depend  on 
mission  requirements  contained  in  the  mis¬ 
sion-essential  subsystem  list  and  the  MNS/ 
ORD  and  the  critical  issues  developed  from 
them  as  defined  in  the  test  plan, 
a.  Operational  Availability  (A0): 

(1)  Operational  availability  covers  all 
segments  of  time  during  which  the  equip¬ 
ment  is  intended  to  be  operational.  Uptime 
includes  operating  time  plus  nonoperating 
(standby)  time  (when  the  equipment  is  as¬ 
sumed  to  be  operable).  Downtime  includes 
preventive  and  corrective  maintenance  and 
associated  administrative  and  logistics  delay 
time. 

A  -  UP  -  _ OT  +  ST _ 

0  OT  +  ST  +  TPM  +  TCM  +  ALDT 

This  relationship  provides  a  realistic  measure 
of  equipment  availability  when  the  equipment 
is  functioning  in  its  operational  environment. 

(2)  One  significant  problem  associated 
with  determining  A0  is  it  becomes  costly  and 
time-consuming  to  define  and  measure  the 
various  parameters.  For  example,  the  proce¬ 
dures  of  AFR  65-110  for  aircraft  availability 
measurement  (FMC  and  PMC)  require  sub¬ 
stantial  time  and  a  large  number  of  test 
assets  in  an  operational  environment  to 
produce  valid  results.  Defining  administra¬ 
tive  and  logistics  downtime  (ALDT)  and  total 
active  preventive  maintenance  time  (TPM) 
under  combat  conditions  is  not  feasible  in 
most  instances.  Nevertheless,  the  operational 
availability  expression  does  provide  an  ac¬ 
cepted  technique  of  relating  standard  reliabil¬ 
ity  and  maintainability  elements  into  a 
mission-oriented  parameter. 

(3)  One  important  aspect  to  note  is  the 


AFOTECP  400-1  15  May  1991 


9-5 


utilization  rate  affect  A0.  The  less  a  system 
is  operated  in  a  given  period,  the  higher  A0 
will  be.  It  is  important,  therefore,  when 
defining  the  total  time  period  to  exclude 
lengthy  periods  during  which  little  or  no 
system  usage  is  anticipated,  such  as  depot 
maintenance  and  nonoperating  storage  time. 
Care  should  also  be  taken  not  to  in¬ 
advertently  exclude  periods  of  time  which 
are  a  part  of  the  operational  environment, 
e.g.,  an  aircraft  sitting  in  an  FMC  status 
over  a  weekend. 

(4)  Another  frequently  encountered  ex¬ 
pression  for  A0  is: 

A  -  MTBM 
0  ~  MTBM  +  MDT 

While  logistics-oriented,  this  form  of  A0 
retains  consideration  of  the  same  basic  ele¬ 
ments.  The  MTBM  and  MDT  intervals  may 
include  corrective  and  preventive  mainte¬ 
nance  and  administrative  and  logistics 
downtime.  This  form  of  the  A0  relationship 
would  generally  prove  more  useful  in  sup¬ 
port  of  early  parameter  definition  and  sensi¬ 
tivity  analysis. 

(5)  A  closely  related  expression  for  Ao  is: 

a  _ _ MTBM _ 

•  “  MTBM  +  MTTRS  +  MLDT  +  MDT^ 

MTBM  can  include  inherent  failures, 
induced  failures,  no-defect  maintenance 
actions,  and  preventive  maintenance  or 
any  combination  of  these. 

MTTRS  includes  preparation  time,  mal¬ 
function  verification  time,  fault  isolation 
defection  time,  repair  time  (replacing, 
repairing,  or  adjusting),  malfunction  final 
test  time,  and  system  final  test  time  if 
applicable. 

MLDT  is  the  average  delay  time  con¬ 
sidering  maintenance  actions  which 
require  parts  and  those  which  do  not. 
It  can  include  base  level  and  depot 
supply  systems. 

The  basic  expression  of  MLDT  is: 

MLDT  =  (A)  (MSRT) 

Where: 

A  =  Percentage  of  corrective 

maintenance  actions  re¬ 
quiring  parts. 

MSRT  =  Mean  supply  response  time 

or  the  weighted  average  of 


response  times  from  base 
stocks  and  from  the  depot 
system. 

MSRT  can  be  calculated  by  the  following 
equation: 

MSRT  =  (C)  (D)  (E)  +  [1-(D)(C)]  (F) 

Where: 

C  =  Percentage  of  parts-re- 

quired  maintenance  actions 
that  are  supported  by 
parts  carried  (allowed)  in 
base  stocks. 

D  =  Percentage  of  allowed 

parts  requirements  normal¬ 
ly  satisfied  from  base 
3tocks. 

E  =  Time  required  to  obtain  a 

part  from  base  stocks. 

F  ss  Mean  requisition  response 

time  (MRRT). 

MDToth#r  is  the  mean  delay  time  for  other 
reasons  such  as  waiting  for  maintenance 
personnel  or  transportation.  It  does  not 
include  supply  delay  time.  Although  several 
miscellaneous  downtimes  can  be  incorporated 
in  this  segment  of  supportability,  such  po¬ 
tential  downtime  should  not  be  considered 
minor  or  insignificant.  One  or  more  defi¬ 
ciencies  in  this  area  can  severely  limit  sup- 
portability  and  hence  A? 

(6)  The  last  expression  for  A0  to  be  pre¬ 
sented  is  based  on  FMC,  MC,  and/or  PMC. 
FMC,  MC,  and  PMC  are  measures  of  A0. 
b.  Achieved  Availability  (A  ): 

(1)  The  definition  of  achieved  availabil¬ 
ity  is  mathematically  expressed  as: 

A  _  OT _ 

*  "  OT  +  TCM  +  TPM 

(2)  Ag  is  frequently  used  during  devel¬ 
opment  testing  and  initial  production  testing 
when  the  system  is  not  operating  in  its 
intended  support  environment.  Excluded 
are  operator  before-and-after  maintenance 
checks  and  standby,  supply,  and  administra¬ 
tive  waiting  periods.  Aa  is  much  more  a 
hardware-oriented  measure  than  is  opera¬ 
tional  availability,  which  considers  operating 
environment  factors.  It  is,  however,  depend¬ 
ent  on  the  preventive  maintenance  policy, 
which  may  be  greatly  influenced  by  non- 
hardware  considerations. 
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c.  Inherent  Availability  (A1): 

( 1)  Under  certain  conditions,  the  logistics 
evaluator  may  find  it  necessary  to  define 
system  availability  with  respect  only  to 
operating  time  and  corrective  maintenance. 
Availability  defined  in  this  manner  is  called 
inherent  availability  (A1): 

A  -  MTBF 
1  "  MTBF  +  MTTR 

(2)  Under  these  ideal  conditions,  the 
evaluation  may  ignore  standby  and  delay 
times  associated  with  scheduled  or  preventive 
maintenance,  as  well  as  administrative  and 
logistics  downtime,  no  defect  maintenance, 
and  maintenance  due  to  induced  failures. 

(3)  Inherent  availability  is  useful  in 
determining  basic  system  operational  char¬ 
acteristics  under  ideal  conditions.  Inherent 
availability  can  also  describe  combined  relia¬ 
bility  and  maintainability  characteristics  or 
define  one  in  terms  of  the  other  during  early 
conceptual  phases  of  a  program  when,  gener¬ 
ally,  these  terms  cannot  be  defined  individ¬ 
ually.  Since  this  definition  of  availability  is 
easily  measured,  it  is  frequently  used  as  a 
contract-specified  requirement. 

9-6.  A  General  Approach  for  Evaluating 
Availability.  The  following  paragraphs 
present  a  general  approach  for  evaluating 
system  availability.  It  is  important  to  note 
for  such  an  analysis  to  be  meaningful  to  an 
equipment  user  or  developer,  it  must  reflect 
the  peculiarities  of  the  system  being  con¬ 
sidered.  The  general  procedures  are: 

a.  The  operational  and  maintenance  con¬ 
cepts  associated  with  system  use  must  be 
defined  in  detail  using  terminology  compat¬ 
ible  with  the  users. 

b.  Using  the  definitions  from  paragraph 
9-4,  construct  a  time  line  availability  model 
which  reflects  the  mission-availability  param¬ 
eters.  Figure  9-2  displays  elements  of  availa¬ 
bility  frequently  included  in  a  quantitative 
assessment  of  availability.  The  up  or  down 
status  of  a  specific  system  during  preventive 
maintenance  must  be  closely  examined. 
Generally,  a  portion  of  the  preventive  main¬ 
tenance  period  may  be  considered  as  uptime. 
Cold  standby  time  must  also  be  examined 
closely  before  determining  system  up  or  down 
status  during  this  period.  The  time  line 
availability  model  may  also  be  constructed 
using  other  commonly  used  reliability  and 
maintainability  parameters.  Figure  9-3 
illustrates  another  approach  to  an  availability 
time  line  model. 

c.  With  the  aid  of  the  time  line  model, 
determine  which  time  elements  represent 


"uptime’'  and  "downtime."  Do  not  be  misled 
by  the  apparent  simplicity  of  this  task.  For 
example,  the  maintenance  concept  may  be 
defined  so  that  the  equipment  must  be 
maintained  in  a  committable  state  during  the 
performance  of  preventive  maintenance. 

d.  Determine  quantitative  values  for  the 
individual  time  elements  of  the  time  line 
models.  Coordinate  these  values  with  the 
user,  developer,  and  contractor. 

e.  Compute  and  track  availability  using 
the  definitions  of  availability  appropriate  for 
the  current  stage  of  system  development. 

f.  Continue  to  check  availability  model 
status  and  update  the  model  as  required. 
Give  special  attention  to  updating  the  model 
as  the  operational,  maintenance,  and  logistics 
support  concepts  mature. 

9-7.  Recovery  Time  Considerations  in 
System  Availability: 

a.  Normally,  availability  measures  imply 
every  hour  has  equal  value  from  the  stand¬ 
point  of  operations  and  the  performance  of 
maintenance  and  logistics  activities.  Addi¬ 
tionally,  the  operational  concept  requires  the 
system  to  function  for  selected  periods.  The 
remaining  time  is  traditionally  referred  to  as 
"off-time,"  during  which  no  activity  is  con¬ 
ducted. 

b.  An  alternative  to  the  "off-time”  or  "cold 
standby"  concepts  is  the  use  of  the  term 
"recovery  time"  (RT)  as  depicted  in  figure 
9-4. 

c.  RT  represents  an  interval  of  time 
during  which  the  system  may  be  up  or  down. 
RT  does  not  appear  in  the  availability  calcu¬ 
lation,  which  is  based  only  on  the  TT  period. 
Take  special  note  of  the  fact  total  active 
corrective  maintenance  time  (TCM— mainte¬ 
nance  required  to  keep  the  system  in  a 
mission  ready  or  available  status)  is  found  in 
both  TT  and  RT.  Corrective  maintenance 
performed  during  the  RT  period  generally  ad¬ 
dresses  hardware  malfunctions  which  do  not 
result  in  a  non-mission-ready  status. 

d.  The  principal  advantage  of  using  the 
"recovery  time”  analysis  is  it  can  provide  a 
more  meaningful  availability  assessment  for 
systems  whose  periods  of  required  availability 
are  predictable  and  whose  preventive  mainte¬ 
nance  constitutes  a  significant  but  delayable 
portion  of  the  maintenance  burden. 

9-8.  Availability  for  Multimission  Sys¬ 
tems.  For  many  modern  weapon  systems, 
availability  is  not  simply  an  up  or  down 
condition.  Systems  such  as  aircraft  have 
multimission/mode  capabilities  and  thus 
require  detailed  techniques  to  characterize 
the  associated  availability  states.  The  defini- 
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Figure  9-3.  Operational  Availability  Time  Line  Model 
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Figure  9-4.  Mission  Availability  Time  Line  Model  (Recovery  Time  Format) 


tion  of  terms,  modes,  and  states  is  especially 
important  in  the  analysis  of  these  complex 
systems.  Generally,  each  mission/mode  will 
require  a  separate  time  line  model. 

9-9.  Data  Analysis.  Much  of  the  data 
required  to  evaluate  availability  comes  from 
the  results  of  the  reliability,  maintainability, 
and  logistics  supportability  evaluations. 
Initial  estimates  of  availability  are  usually 
established  from  contractor  predictions  and 
updated  by  the  test  team  during  test  execu¬ 
tion.  Sensitivity  of  availability  to  changes  in 
reliability,  maintainability,  and  logistics 
supportability  parameters  at  the  subsystem 
level  is  analyzed  by  incrementally  changing 
MTBMs,  MDTs,  MTBCFs,  delay  times,  etc., 
to  identify  constraints  and  to  determine  if 
one  or  some  combination  of  the  parameters 
significantly  changes  the  availability. 

9-10.  Availability  Simulation/Modeling. 
Paragraph  9-6  discussed  a  general  approach 
for  evaluating  system  availability  using  time 
lines  and  direct  calculation  of  operational, 
achieved,  and  inherent  availability.  Another 
approach  is  to  construct  and  run  a  simula¬ 
tion  model  of  the  system  and  its  support 
structure.  This  model  is  based  on  the  user’s 
operations  and  maintenance  concepts,  user 
definitions  of  availability  and  the  time  line 
analysis  presented  in  paragraph  9-6.  By 
emulating  the  operational  employment  of  the 
system,  a  simulation  model  gives  the  analyst 
a  powerful  tool  to  evaluate  a  system’s  readi¬ 
ness  and  the  factors  that  influence  it. 
a.  Justification  for  Simulation: 

(1)  Availability  is  a  measure  of  the 
degree  to  which  an  item  is  in  the  operable 
and  committable  state  at  the  start  of  the 
mission  when  the  mission  is  called  for  at  a 
random  point  in  time.  For  a  given  utiliza¬ 
tion  rate,  availability  depends  on  the  reliabil¬ 
ity  and  maintainability  characteristics  of  the 


system  and  the  support  posture  of  the  logis¬ 
tics  supportability  factor  involved. 

(2)  Availability  is  not  a  direct  charac¬ 
teristic  of  the  system  but  rather  is  a  result 
of  system  characteristics  such  as  those  men¬ 
tioned  above.  Point  estimates  of  availability, 
calculated  from  current  and  past  observa¬ 
tions,  can  provide  an  overall  measure  of  past 
performance  and  can  be  used  as  a  manage¬ 
ment  indicator.  However,  even  during  OT&E 
of  weapon  systems  in  the  final  stages  of  the 
acquisition  process,  availability  data  generally 
cannot  be  used  alone  to  provide  reasonable 
projections  of  expected  future  system  avail¬ 
ability. 

(3)  Early  in  the  acquisition  phase,  before 
the  production  decision,  systems  under  test 
do  not  reside  in  an  operationally  representa¬ 
tive  environment.  Additionally,  the  support 
system  (manpower,  test  equipment,  technical 
manuals,  etc.)  is  not  representative  of  the 
one  the  Air  Force  will  use  when  the  weapon 
system  is  fielded.  In  short,  predictions  of 
expected  future  availability  must  be  made  in 
the  context  of  expected  future  system  charac¬ 
teristics  and  scenarios.  The  predictions  are 
continuously  refined  as  the  system  matures 
and  early  test  data  are  replaced  by  findings 
in  more  operationally  representative  environ¬ 
ments. 

(4)  The  AFOTEC  Logistics  Studies  and 
Analysis  Division  (LG4)  uses  computer  simu¬ 
lation  models  to  estimate  future  availability 
of  weapon  systems  when  maintained  and 
operated  in  accordance  with  the  using  com¬ 
mand’s  maintenance  and  operational  concepts 
in  peace  and  war.  Models  allow  for  exten¬ 
sive  "what  if”  questions  and  help  quantify 
impacts  of  proposed  changes  to  the  system. 
With  proper  information  regarding  expected 
system  availability,  decision  makers  can  call 
for  changes  in  the  operational  and  mainte¬ 
nance  concepts,  manpower  levels,  spare  part 
policy,  etc.,  to  accommodate  the  unique 
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requirements  of  the  system. 

(5)  AFOTEC  uses  several  computer  lan¬ 
guages  to  develop  simulation  models.  The 
major  languages  are  the  simulation  language 
for  alternative  modeling  (SLAM)  and  logis¬ 
tics  composite  model  (LCOM).  SLAM  is  a 
high-level  simulation  language  and  is  used 
extensively  by  HQ  AFOTEC/LG4.  The 
LCOM  simulation  package  is  a  standard  Air 
Force  data  system  and  is  used  extensively  by 
the  manpower  community. 

b.  Modeling  Methods  and  Tools.  If 
the  decision  has  been  made  to  use  simulation 
modeling  for  analysis  of  system  availability, 
it  is  generally  more  cost-  and  time-effective 
to  use  existing  models  such  as  LCOM  and 
dedicated  modeling  languages  like  SLAM 
rather  than  to  develop  a  model  code  from 
scratch.  The  analyst  will  be  concerned  with 
simulating  system  failure  and  repair  charac¬ 
teristics  as  well  as  activity  flow  and  day-to- 
day  operations  in  a  clearly  defined  scenario 
or  operational  situation.  Selection  of  simula¬ 
tion  tools  is  dependent  on  the  system  under 
evaluation,  the  computer  system  to  be  used, 
and  the  time  and  resource  available  for 
model  development  and  analysis.  Each  of 
the  tools  offers  particular  advantages  for 
particular  applications. 

( 1)  SLAM  combines  process-oriented  capa¬ 
bilities  with  continuous  simulation  and  event 
scanning  capabilities.  SLAM  is  FORTRAN- 
based  and  allows  the  modeler  to  use 
FORTRAN  subroutines  to  enhance  SLAM 
capabilities.  SLAM  provides  a  set  of  stand¬ 
ard  subprograms  that  the  modeler  can  use  to 
schedule  events,  manipulate  files,  collect 
statistics,  and  generate  random  samples. 

(2)  The  LCOM  was  designed  to  model 
aircraft  operations  and  support  functions  at 
base-  or  wing-level.  LOOM  may  be  used  to 
model  a  system  other  than  aircraft  operation 
and  support,  but  the  system  must  be  analo¬ 
gous  to  an  air  base  operation.  Since  the 
logic  used  to  describe  aircraft  maintenance 
is  similar  to  many  other  process-oriented 
operations,  LCOM  can  be  applied  to  a  fairly 
large  number  of  systems,  e.g.,  the  Space 
Transportation  System.  Analysts  competent 
in  the  vise  of  LCOM  can  modify  and  apply  it 
to  any  situation  or  environment  that  can  be 
represented  as  a  network  using  tasks  which 
require  time  and/or  resources  for  accomplish¬ 
ment. 

c.  Model  Level  of  Detail  Require¬ 
ments: 

(1)  The  level  of  detail  a  simulation  model 
requires  depends  on  the  system  under  test 
and  the  time  and  resources  available  for 
model  development.  Generally,  items  in  the 
early  phases  of  acquisition  cannot  and  should 


not  be  modeled  to  a  great  level  of  detail, 
since  maintenance  processes  and  policies 
have  not  been  fully  established.  For  more 
complex  and  sophisticated  items,  modeling  of 
subcomponents  may  need  to  be  accomplished 
at  the  major  component  level  because  of  lack 
of  data  and  maintenance  concepts.  For  more 
mature  systems,  the  modeler  may  have  a 
wealth  of  information  on  existing  piece  parts 
and  maintenance  processes.  In  this  case,  a 
more  detailed  model  may  be  desired. 

(2)  As  a  minimum,  the  model  should 
consider  the  factors  shown  in  figure  9-5. 
Each  of  the  factors  or  parameters  may  be 
specified  in  the  model  at  various  levels  of 
detail,  and  it  is  incumbent  on  the  model 
developer  to  correctly  specify  the  level  of 
detail  LAW  the  test  requirements  and  the 
time  and  resources  available. 

d.  Availability  Measures  of  Effec¬ 
tiveness  (MOE)  in  Modeling.  Simulation 
models  can  calculate  availability  measures  in 
many  ways.  A  great  deal  of  car  :  is  needed 
to  ensure  the  model  output  is  vhat  is  re¬ 
quired  or  desired.  It  is  somewaat  difficult 
to  obtain  "nonstandard"  outputs  from  LCOM. 
However,  SLAM  allows  the  analyst  to  care¬ 
fully  define  the  desired  measures  and  to 
ensure  the  model  is  correctly  accounting  for 
these  measures  during  model  execution. 

e.  Obtaining  Model  Input  Data: 

(1)  The  amount  of  input  data  required  for 
the  execution  of  the  model  can  be  signifi¬ 
cant,  depending  on  the  level  of  model  detail. 
It  cannot  be  emphasized  enough  that  the 
quality  of  the  input  data  is  directly  related 
to  the  amount  of  confidence  one  can  place  on 
the  resulting  availability  estimates. 

(2)  In  most  system  evaluations,  relia¬ 
bility  characteristics  will  be  the  most  impor¬ 
tant  factor  influencing  availability.  Early  in 
the  procurement  and  testing  process,  the 
analyst  may  have  to  rely  on  the  development 
contractor  for  reliability  estimates  of  the 
system  components.  However,  the  analyst 
may  obtain  independent  estimates  of  reliabil¬ 
ity  parameters  from  comparability  analysis 
and/or  independent  research  and  judgment. 
After  deployment  of  the  system,  the  analyst 
may  use  actual  failure  history  data  and  test 
results. 

(3)  Similarly,  maintainability  data  may 
initially  be  obtained  from  the  contractor.  In 
addition,  the  analyst  must  work  with  experi¬ 
enced  maintenance  personnel  or  personnel 
familiar  with  similar  systems  to  further 
estimate  maintenance  times  and  to  determine 
maintenance  processes  (i.e.,  networks).  As 
testing  proceeds,  actual  measured  data  may 
be  used  to  update  preliminary  data  values. 

(4)  The  system  operational  and  main- 
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o  Reliability  parameters 

Mean  time  between  downing  events  (or  MTBM)  for  components  and/or  system 
(including  dormant  reliability  if  applicable) 

Operating  time  characteristics  in  various  states 

o  Maintainability  parameters 

Setup/fault  isolation/remove/repair/checkout  times  for  components 
BIT/FIT  effectiveness  parameters 
o  Logistics  supportability  parameters 

Support  equipment  requirements 
Facility  requirements 

Repair  part  stockage  policies  (onhand  inventories,  WRSK,  POS,  and  cannibalization 
policy) 

Supply  characteristics,  including  transportation  delays  between  field  and  depot 
o  Maintenance  policy  parameters. 

Scheduled  maintenance  periods 
Level  of  repair  considerations 
o  Scenario  and  conditions 
Peacetime  conditions 
Wartime  mission  definition 


Figure  9-5.  Factors  to  Include  in  Availability  Modeling 


tenance  concepts  can  be  used  for  setting 
maintenance  policies,  defining  mission  scenar¬ 
ios,  and  determining  manpower,  facilities, 
and  support  equipment  characteristics. 
Coordination  meetings  should  be  held  with 
the  appropriate  using  command  personnel  to 
keep  abreast  with  current  concepts. 

£  Example:  F-15E  Availability  Model. 
The  F-15E  availability  model  is  an  analysis 
tool  used  in  the  operational  suitability  evalu¬ 
ation  of  the  F-15E  during  combined  develop¬ 
mental  test  and  evaluation  and  operational 
test  and  evaluation  (DT&E/OT&E)  and  the 
dedicated  operational  test  and  evaluation 
(OT&E)  phases.  The  model  is  used  to  evalu¬ 
ate  the  availability,  mission  reliability,  and 
maintainability  of  a  mature  F-15E  squadron 
during  various  scenarios.  Availability  is 
measured  by  mission  capable  (MC)  rate  and 


sortie  generation  rate  (SGR).  Mission  relia¬ 
bility  is  measured  by  break  rate  (BR). 
Maintainability  is  measured  by  fix  rate  (FR), 
mean  repair  time  (MRT),  and  manpower 
spaces  per  aircraft  (SPA).  Figure  9*6  depicts 
the  F-15E  model  flow. 

(1)  The  model  was  built  to  describe  the 
major  aspects  of  the  HQ  TAC  operational 
environment.  Similar  flying  schedules,  sortie 
length,  maintenance  priorities,  maintenance 
concepts,  resource  allocation  per  Air  Force 
specialty  code  (AFSC),  resource  usage  per 
AFSC,  and  scheduled  and  unscheduled  main¬ 
tenance  per  four-digit  work  unit  code  (WUC) 
are  included.  Mean  times  between  mainte¬ 
nance  (MTBM)  and  maintenance  task  times 
for  subsystems  are  inputs  to  the  model. 
Unknowns  in  the  model,  such  as  what  time 
items  fail,  are  calculated  in  the  model  using 
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MTBM  data  and  probabilities. 

(2)  Manpower  resources  are  specified  by 
AFSC  and  in  some  cases  by  skill  level.  The 
F-15E  manpower  allocations  and  usage  follow 
the  intended  maintenance  concept  (rivet 
workforce).  Quantities  of  maintenance  per¬ 
sonnel  of  a  certain  AFSC  can  be  decreased 
or  increased  to  determine  the  effect  on  F-15E 
availability.  Many  other  items,  such  as 
support  equipment  availability,  spares  levels, 
MTBM,  and  task  times,  may  be  varied  to 
answer  "what  if’  questions  and  perform 
sensitivity  analyses. 

(3)  The  model  operates  by  beginning  the 
simulation  at  time  =  0,  and  permitting 
scheduled  events  such  as  scheduled  mainte¬ 
nance  and  flying  sorties  to  occur.  After  each 
sortie,  the  failure  clocks  for  each  WUC  are 
checked  to  see  if  a  failure  occurred  on  that 
sortie.  If  a  failure  did  occur,  the  necessary 
organizational-level  maintenance  and  thru- 
flight  maintenance  are  performed  and  the 
aircraft  is  ready  to  fly  again.  Shop  mainte¬ 
nance  is  also  begun  on  removed  LRUs,  if 
needed.  The  simulation  continues  until  the 
end  of  the  simulation  time  is  reached.  The 
model  actually  "simulates"  what  occurs  at  a 
TAC  flying  squadron. 

(4)  A  random  number  generator  is  used 
to  obtain  random  samples  from  a  specific 
statistical  distribution:  triangular,  normal, 
exponential,  or  lognormal.  For  each  simula¬ 
tion  of  a  scenario,  at  least  five  different 
random  number  seeds  are  used  for  the  ran¬ 
dom  number  generator.  The  five  simulation 
results  are  averaged  to  obtain  the  "final" 
results. 

9-11.  Lessons  Learned: 

a.  Studies  of  major  weapon  systems  during 
test  and  evaluation  have  proven  to  be  ex¬ 
tremely  labor  intensive,  requiring  interaction 
among  the  model  builder  and  the  system 
developers,  users,  testers,  and  supporters.  To 
provide  timely,  useful  analyses  in  a  com¬ 
pressed  acquisition/test  environment,  HQ 
AFOTEC/LG4  initiated  model  committees  to 
support  model  development  and  studies.  To 
better  implement  the  intent  of  Director, 
Operational  Test  and  Evaluation  (DOT&E) 
policy,  HQ  AF0TEC/LG4  directed  a  commit¬ 
tee  approach  to  modeling/simulation  (M/S) 
application,  development,  and  documentation. 
An  M/S  committee,  composed  of  the  LG4  M/S 
technical  advisor,  the  primary  analyst  of  the 
system,  and  additional  analysts,  will  be 
formed  for  each  M/S  effort  designated  by 
LG4.  Primary  objectives  of  each  M/S  com¬ 
mittee  are  to  provide  a  forum  for  review  and 
for  guiding  steps  taken  during  the  M/S 
development  effort  to  establish  M/S  credibil¬ 


ity,  to  provide  a  forum  for  exchanging  M/'S 
ideas  and  techniques  between  analysts,  to 
ensure  a  verified  and  validated  M/S  product, 
and  to  recommend  M/S  documentation  for 
release  to  outside  agencies.  In  addition,  the 
committee  review  structure  is  intended  to 
maintain  independence  between  M/S  develop¬ 
ment  and  its  use  to  support  OT&E. 

b.  Timely  collection  of  data  for  subsystem 
availability  evaluation  must  be  addressed 
during  advance  planning.  The  analyst  must 
determine  data  requirements  and  assess  the 
best  method  of  acquiring  the  data.  If  the 
test  will  be  run  at  a  contractor  facility  with 
the  contractor  performing  the  maintenance, 
specific  preparation  is  vital.  The  contractor 
should  be  required  to  collect,  document,  and 
provide  all  data  necessary  to  perform  a 
complete  system  analysis. 

c.  The  maturity  of  the  system  will  in¬ 
fluence  the  emphasis  given  to  an  availabil¬ 
ity  evaluation.  An  example  of  the  changing 
emphasis  is  the  F-16  development.  During 
IOT&E  with  prototype  aircraft,  availability 
was  not  assessed,  but  specific  hardware 
reliability  and  maintainability  deficiencies 
were  reported  for  correction.  During  OT&E, 
using  full-scale  development  aircraft,  availa¬ 
bility  was  determined  using  definitions  and 
procedures  contained  in  the  test  plan.  Final¬ 
ly,  during  multinational  OT&E  (MOT&E), 
AFR  65-110  procedures  were  applied  to  the 
fleet  of  operational  aircraft  to  measure  FMC, 
PMC,  and  other  availability  rates. 

d.  The  test  environment  can  have  a  major 
effect  on  the  evaluation  of  system  availabil¬ 
ity.  During  IOT&E,  and  particularly  during 
combined  DT&E/IOT&E,  contractor  mainte¬ 
nance  may  be  oriented  toward  completing 
specific  test  events  rather  than  maintaining 
systems  in  a  mission-capable  status.  In 
addition,  the  system  may  be  down  for  ex¬ 
tended  periods  while  undergoing  development 
modifications.  It  is  important,  therefore,  to 
reach  a  common  agreement  with  all  test  par¬ 
ticipants  as  to  what  time  base  will  be  used 
in  the  availability  computations.  Lengthy 
periods  during  which  little  or  no  system 
usage  occurs  should  be  critically  examined  for 
applicability  to  the  system's  availability 
calculations. 

9-12.  Key  References  for  Availability: 

a.  AFR  65-110,  Aerospace  Vehicle  and 
Equipment  Inventory,  Status,  and  Utilization 
Reporting  System  (AVTSURS). 

b.  AFP  57-9,  Defining  Logistics  Require¬ 
ments  in  Statement  of  Operational  Need. 

c.  DOD  3235. 1-H,  Test  and  Evaluation  of 
System  Reliability,  Availability,  and  Main- 
tainability-A  Primer. 
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d.  MIL-STD  721,  Definitions  of  Effective¬ 
ness  Terms  for  Reliability,  Maintainability, 
Human  Factors,  and  Safety. 


e.  HQ  AF0TEC/LG4  Guidelines  for  Model¬ 
ing  and  Simulation  (M/S)  Application.  Devel¬ 
opment,  and  Documentation. 
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Chapter  10 
RELIABILITY 


10-1.  General: 

a.  Reliability  is  a  key  factor  which  in¬ 
fluences  a  system’s  effectiveness,  logistics 
support  requirements,  and  life  cycle  cost. 
Together,  the  reliability  and  maintainability 
characteristics  of  a  system  help  determine 
the  system’s  operational  readiness. 

b.  Reliability  quantitatively  describes  the 
degree  to  which  a  system  is  likely  to  be 
failure-free  during  a  given  period  of  opera¬ 
tion  understated  conditions.  The  ability  to 
express  reliability  numerically  is  important 
because  it  enables  personnel  involved  in  the 
acquisition  process  to  concretely  identify  the 
user’s  needs,  contractual  specifications,  test 
guidelines,  and  performance  assessment. 
When  evaluating  reliability,  OT&E  must  not 
only  address  the  system  in  terms  of  its 
ability  to  complete  the  mission  but  also  in 
terms  of  the  effect  on  the  logistics  system. 

c.  This  chapter  presents  background 
material  on  reliability,  current  Air  Force  and 
DOD  policy  regarding  reliability,  and  an 
approach  for  evaluating  reliability  during 
T&E. 

10-2.  Reliability  Requirements.  AFP 
57-9,  Defining  Logistics  Requirements  in 
Statements  of  Operational  Need,  divides 
reliability  terms  into  two  categories,  opera¬ 
tional  and  contractual. 

a.  Operational.  With  the  implementing 
and  supporting  commands’  assistance,  the 
using  command  develops  reliability  require¬ 
ments  based  on  mission  needs  which  are 
expressed  in  operational  terms.  AFP  57-9 
includes  a  list  of  acceptable  operational 
reliability  terms  and  their  definitions  to  be 
used  in  statements  of  operational  need, 
maintenance  concepts,  program  management 
directives,  and  program  management  plans. 
These  terms  describe  the  required  reliability 
performance  of  the  total  system  and  its 
supporting  elements  when  operating  in  its 
planned  environment.  During  acquisition, 
they  are  used  to  plan  and  manage  the  relia¬ 
bility  program  and  to  evaluate  reliability 
performance.  Operational  reliability  require¬ 
ments  also  provide  the  basis  for  conducting 
OT&E,  and  the  program  decision  authorities 
use  these  to  evaluate  achievement  of  thresh¬ 
olds  and  mature  reliability  performance. 
Therefore,  it  is  imperative  that  OT&E  test 
plans  reflect  the  operational  requirements 
and  terminology  expressed  in  the  operational 
and  support  planning  documents. 


b.  Cc  tractual: 

(1)  T  e  implementing  command  develops 
contracti  .1  reliability  and  maintainability 
terms  (g  lerally  limited  to  events  which  are 
subject  t  control  by  the  contractor)  from  the 
operatior  .1  terms  and  incorporates  them  into 
contracts  The  prediction  and  measurement 
of  contr  :t  terms  do  not  always  provide 
values  t  at  can  be  directly  compared  to 
operatior  .1  terms.  Therefore,  the  imple¬ 
menting  ommand  must  clearly  identify  each 
contracti  .1  term  to  prevent  any  confusion 
with  sir  lar  operational  terms  and  must 
establish  in  audit  trail  to  relate  these  terms. 

(2)  E  ch  weapon  system  and  equipment 
acquisiti'  l  authorized  by  a  specific  program 
manager  rnt  directive  (PMD)  is  required  to 
have  a  r  liability  and  maintainability  man¬ 
agement  ’lan  (RMMP).  The  RMMP  will  be 
updated  nnually  or  as  significant  program 
changes  ccur.  It  is  a  description  of  the 
reliabilit  and  maintainability  (R&M)  pro¬ 
gram  for  achieving  the  user’s  R&M  require¬ 
ments  a  i  thresholds.  The  plan  provides 
paramet-  translation  methodologies  used  to 
translate  -he  operational  R&M  requirements 
into  mt  surable,  enforceable  contractual 
requirexr  nts  (including  test  terms). 

10-3.  D  fruitions  and  Concepts.  Stand¬ 
ard  tern  rology  is  vital  to  good  communica¬ 
tions  bet  een  all  participants  involved  in  a 
system’s  cquisition.  Lack  of  standard  termi¬ 
nology  h  s  traditionally  caused  many  prob¬ 
lems.  ’  le  primary  objective  of  standard 
terminol  »y  is  to  have  the  users,  developers, 
testers,  ad  supporters  all  use  a  common 
baseline  >r  assessment  of  reliability  perform¬ 
ance.  R  iability  terms  are  defined  in  MIL- 
STD  72)  the  Multiservice  Memorandum  of 
Agreeme  t  (MOA)  for  OT&E,  and  other 
sources  <  ee  attachment  5).  Important  relia¬ 
bility  cor  :epts  are  clarified  below: 
a.  Re  iability: 

(1)  F  liability  is  defined  as  the  proba¬ 
bility  ai  item  can  perform  its  intended 
function  it  a  specified  interval  under  stated 
conditior  .  Reliability  is  also  defined  as  the 
duration  or  probability  of  failure-free  per¬ 
formance  under  stated  conditions. 

(2)  I  a  system  is  capable  of  performing 
multiple  nissions  or  if  it  can  perform  one  or 
more  of  ts  missions  while  operating  in  a 
degradec  condition,  the  concept  of  a  unique 
mission  ^liability  becomes  difficult  to  define. 
In  such  ises,  it  is  preferable  to  use  a  relia- 
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bility  measure  that  is  not  based  solely  on  the 
length  of  a  specified  time  interval  but  rather 
on  the  definition  of  a  specific  mission  profile 
or  set  of  profiles. 

(3)  The  meaning  of  the  terms  "stated 
conditions"  and  "specified  interval"  are  impor¬ 
tant  to  the  understanding  of  reliability.  The 
term  "stated  conditions"  refers  to  the  com¬ 
plete  definition  of  the  scenario  in  which  the 
system  will  operate,  be  maintained,  and  be 
supported.  For  aircraft,  operating  conditions 
may  include  park,  taxi,  takeoff,  climb,  cruise, 
air-refuel,  air  combat,  and  other  related 
activities.  For  strategic  missiles,  the  condi¬ 
tions  would  include  the  alert  posture.  What¬ 
ever  the  system,  the  stated  conditions  should 
reflect  operational  use.  The  term  "specified 
interval"  refers  to  the  length  of  the  mission 
described  in  a  mission  profile.  This  interval 
may  include  multiple  factors.  For  example, 
for  a  fighter  aircraft,  an  air-to-air  combat 
engagement  profile  will  define  an  interval 
containing  X  hours  of  radar  on  time,  Y 
rounds  fired,  and  Z  hours  flown.  The  speci¬ 
fied  interval  is  a  segment  of  the  mission 
profile  which  requires  certain  activity  to  take 
place  using  equipment  to  accomplish  that 
activity. 

b.  Reliability  Incident  Classification: 

(1)  System  Failures.  System  failures  are 
hardware  malfunctions  or  software  errors. 
They  may  or  may  not  affect  the  mission’s 
essential  functions,  and  they  may  or  may  not 
require  spares  for  correction. 

(2)  Mission  Failures.  Mission  failures 
are  the  loss  of  any  of  the  mission’s  essential 
functions.  System  hardware  failures,  soft¬ 
ware  errors,  operator  errors,  and  errors  in 
technical  orders  that  cause  such  a  loss  are 
included  in  this  category. 

(3)  Inherent  Failure.  The  item  can  no 
longer  meet  the  minimum  specified  perform¬ 
ance  because  of  failure  resulting  from  inter¬ 
nal  design  and  manufacturing  characteristics. 

(4)  Induced  Failure.  The  item  can  no 
longer  meet  the  minimum  specified  perform¬ 
ance  because  of  some  induced  condition  and 
not  because  of  its  own  internal  failure  pat¬ 
tern. 

(5)  No-Defect  Maintenance  Action.  Main¬ 
tenance  resources  were  expended  because  of 
policy,  modification,  location,  or  cannibaliza¬ 
tion,  and  no  defect  was  identified  at  the  time 
of  maintenance. 

(6)  System/Mission  Failures  Requiring 
Spares.  Failures  of  system/mission  essential 
equipment  that  require  spares  for  correction. 

(7)  Unscheduled  Spares  Demand.  All 
unscheduled  spares  demands  require  a  re¬ 
sponse  from  the  supply  system,  so  they  form 
the  basis  for  evaluating  supply-related  relia¬ 


bility. 

c.  Chargeable/Nonchargeable  Fail¬ 
ures. 

(1)  Contract  requirements  are  often 
established  for  the  subset  of  mission  failures 
and/or  system  failures  for  which  the  contrac¬ 
tor  can  be  held  accountable.  Normally 
excluded  from  contractual  chargeability  are 
such  failure  categories  as  operator  or  mainte¬ 
nance  errors,  item  abuse,  secondary  failures 
caused  by  another  primary  failure,  and 
failures  for  which  a  "fix"  has  been  identified 
but  not  incorporated  in  the  test  article  that 
failed. 

(2)  In  operation,  all  failures  (in  fact,  all 
unscheduled  maintenance  actions)  are  rele¬ 
vant  regardless  of  contractual  chargeability 
and  should  be  included  in  operational  evalua¬ 
tions.  One  exception  is  failures  caused  by 
test-unique  hardware,  software,  procedures, 
etc.,  are  generally  nonrelevant  (i.e.,  non- 
chargeable)  to  the  contractual  (DT&E)  and 
operational  (OT&E)  evaluations. 

cL  Mean  Time  Between  Failure 
(MTBF).  MTBF  is  a  contractual  term  de¬ 
fined  as  the  total  functioning  life  of  a  popula¬ 
tion  of  an  item  during  a  specific  measure¬ 
ment  interval  divided  by  the  total  number  of 
failures  within  the  population  during  the 
interval.  MTBF  can  be  interpreted  as  the 
expected  length  of  time  a  system  will  be 
operational  between  failures.  The  definition 
is  true  for  time,  cycles,  flying  hours,  or  other 
measure-of-Iife  units.  These  various  meas- 
ure-of-life  units  permit  the  MTBF  term  to  be 
tailored  to  the  reliability  requirements  of  a 
specific  system.  When  MTBF  is  specified  as 
a  constant,  it  is  based  on  the  assumption 
the  underlying  failure  distribution  is  expo¬ 
nential.  Under  this  assumption,  the  proba¬ 
bility  a  system  will  operate  without  failure 
for  time  t  (i.e.,  its  reliability)  is  R(t)  =  e'1/MTBF. 

e.  Failure  Rate.  The  number  of  failures 
of  an  item  per  measure-of-life  unit  (e.g., 
cycles,  time,  miles,  or  events  as  applicable). 
The  failure  rate  is  the  reciprocal  of  the 
MTBF. 

f.  Hardware  Failures  Versus  Software 
Failures  (Errors): 

(1)  When  a  system  that  incorporates 
hardware  and  software  is  undergoing  OT&E, 
there  will  be  failures  because  of  hardware 
components  and/or  computer  programs  (soft¬ 
ware).  Software  failures  have  the  same  net 
effect  on  system  performance  as  hardware 
failures;  therefore,  a  need  exists  to  deter¬ 
mine  the  "reliability"  of  the  software  and  its 
impact  on  system  availability. 

(2)  Some  basic  definitions  in  this  discus¬ 
sion  are: 

(a)  Failure.  The  inability  of  a  system 
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or  system  component  to  perform  a  required 
function  within  specified  limits.  A  software 
failure  may  be  produced  when  a  software 
fault  is  encountered. 

(b)  Software  Fault.  A  manifestation  of 
an  error  in  software  which,  when  encoun¬ 
tered,  may  cause  a  software  failure. 

(c)  Software  Error.  A  human  action 
which  results  in  software  containing  a  fault. 
Examples  include  omission  or  misinterpreta¬ 
tion  of  user  requirements,  incorrect  transla¬ 
tion  or  omission  of  a  specification  require¬ 
ment,  etc.  (IEEE  STD  729-1983). 

(3)  A  basic  difference  between  hardware 
and  software  is  that  hardware  fails  periodi¬ 
cally  over  time,  software  does  not.  Hardware 
failures  which  have  been  repaired  can  recur, 
and  a  failure  rate  can  be  established  for  the 
particular  item.  In  contrast,  software  does 
not  degrade.  A  perfect  computer  program 
would  not  deteriorate  and  would  remain 
failure  free  throughout  its  useful  life.  Soft¬ 
ware  faults  which  cause  system  failures  were 
built  into  the  program  but  not  previously 
detected.  However,  once  a  particular  soft¬ 
ware  fault  has  been  discovered  and  corrected, 
it  will  not  occur  again.  This  is  not  to  say 
the  software  is  reliable.  As  a  result  of  the 
correction  process,  additional  software  errors 
may  be  introduced  into  the  system  (a  concept 
referred  to  as  "imperfect  debugging"). 

(4)  Current  test  and  evaluation  meth¬ 
odologies  assume  all  software  faults  will  be 
fixed  by  system  maturity.  System  reliability 
is  therefore  projected  solely  on  the  basis  of 
hardware  failures.  An  erroneous  picture  of 
system  reliability  can  thus  be  presented  to 
the  user  and  senior  decision  makers.  Relia¬ 
bility  assessments  and  projections  directly 
feed  into  equations  of  operational  availability. 
An  erroneous  reliability  will  yield  an  errone¬ 
ous  availability. 

(5)  Software  reliability  is  the  probability 
that  software  will  not  cause  the  failure  of  a 
system  for  a  specified  time  under  specified 
conditions.  The  probability  is  a  function  of 
the  inputs  to  and  use  of  the  system  rather 
than  a  function  of  the  existence  of  faults  in 
the  software.  The  inputs  to  the  system 
determine  whether  existing  faults,  if  any,  are 
encountered. 

(6)  HQ  AFOTEC  has  developed  a  method 
to  determine  the  effects  of  software  on  sys¬ 
tem  reliability.  Software  maturity  data 
gathered  during  developmental  and  operation¬ 
al  testing  are  used  as  input  to  the  software 
failure  rate  model.  The  effects  from  software 
enhancements  developed  during  block  release 
cycles  and  fault  introduction  through  error 
correction  are  added  to  give  a  comprehensive 
yet  practical  measure  of  software  reliability. 


An  estimate  of  the  total  number  of  faults  in 
a  software  system  is  determined  from  the 
failure  rate,  and  a  software  mean  time  be¬ 
tween  critical  failure  <MTBCF)  is  defined. 

(7)  Even  though  there  is  a  basic  differ¬ 
ence  between  hardware  and  software  failure, 
the  net  effect  on  system  performance  is 
essentially  the  same.  A  method  for  assess¬ 
ing  and  predicting  the  software’s  effects  on 
system  reliability  will  provide  decision 
makers  with  a  better  understanding  of  the 
operational  suitability  of  weapon  systems 
entering  the  inventory.  A  true  system  relia¬ 
bility  picture  at  system  maturity  can  be 
given  by  providing  an  expected  number  of 
critical  software  failures  as  well  as  a  mean 
time  between  each  critical  system  failure 
caused  by  software.  Thus,  the  evaluator 
must  consider  the  effects  of  software  failures 
in  the  final  assessment  of  a  system/ 
equipment’s  reliability  and  the  impact  on 
effectiveness  and  availability.  (For  a  further 
discussion  of  AFOTEC’s  approach  to  the 
evaluation  of  software,  see  chapter  12,  Soft¬ 
ware  Evaluation,  of  this  pamphlet  and 
AFOTECP  800-2,  Software  Operational  Test 
and  Evaluation  Guidelines.) 

10-4.  R&M  Policy  Guidance.  AFOTEC 
responsibilities  (AFR  800-18): 

a.  Assign  an  R&M  OPR  at  AFOTEC 
headquarters.  (That  OPR  is  the  Logistics 
Studies  and  Analysis  Division  (HQ  AFOTEC/ 
LG4).) 

b.  Develop  reliability  test  objectives,  meth¬ 
odology,  data  requirements,  evaluation  crite¬ 
ria,  and  analytical  techniques  to  include  in 
AFOTEC  OT&E  test  plans.  Approve  these 
same  elements  in  Air  Force-directed, 
MAJCOM-conducted  OT&E  test  plans. 

c.  Review  program  documentation  (such 
as  operational  and  support  concept  docu¬ 
ments,  decision  coordinating  papers,  and 
program  management  directives  (PMD))  for 
the  adequacy  of  reliability  parameters  for 
measurement  and  evaluation  during  OT&E. 
Provide  inputs  into  program  documentation 
for  assessment  of  these  reliability  parameters 
during  OT&E. 

d.  Develop  methods,  policy,  and  proce¬ 
dures  for  evaluating  R&M  during  OT&E  and 
provide  guidance  to  other  OT&E  agencies. 

e.  Plan,  conduct,  monitor,  and  report  the 
results  of  logistics  assessments  performed 
during  OT&E  of  systems  and  equipment. 
This  assessment  includes  reliability  and 
maintainability  evaluations  of  PMD  thresh¬ 
olds  and  the  use  of  reliability  data  in  eval¬ 
uating  logistics  supportability. 

f.  Identify  the  estimated  OT&E  data 
requirements  to  the  implementing  command 
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in  time  to  allow  for  adequate  funding. 
Provide  OT&E  data  requirements  to  the 
implementing  command  to  include  in  con¬ 
tracts. 

g.  Help  the  implementing  command, 
AFLC,  and  HQ  USAF  develop  and  implement 
data  systems  for  verification  of  reliability 
performance  during  DT&E,  assessment  dur¬ 
ing  OT&E,  and  measurement  of  product 
performance  throughout  the  system's  life 
cycle. 

h.  Provide  ATC  with  OT&E  philosophy, 
policy,  and  experience  to  develop  and  improve 
reliability  engineering  education  and  training 
programs  and  courses.  Conduct  reliability 
training  courses  or  seminars,  as  required. 

10-5.  System  Reliability  Design  Objec¬ 
tives.  There  are  two  very  different  system 
reliability  design  objectives.  One  is  to  en¬ 
hance  system  effectiveness;  the  other  is  to 
minimize  the  burden  of  owning  and  operating 
the  system.  The  first  objective  is  addressed 
by  means  of  mission  reliability,  the  second  by 
means  of  logistics-related  reliability.  Meas¬ 
ures  of  mission  reliability  address  only  those 
incidents  that  affect  mission  accomplishment. 
Measures  of  logistics-related  reliability  ad¬ 
dress  all  incidents  that  require  a  response 
from  the  logistics  system. 

a.  Mission  Reliability.  The  probability 
a  system  will  give  a  specified  performance  for 
the  duration  of  a  mission  when  used  in  the 
manner  and  for  the  purpose  intended,  given 
the  system  is  functioning  properly  at  the 
start  of  the  mission.  The  following  terms 
relate  to  mission  reliability: 

(1)  Aircraft  Abort  Rates.  Often  used  to 
assess  aircraft  mission  reliability  and  include 
before  flight  abort  (BFA)  rate,  in-flight  abort 
(IFA)  rate,  and  total  abort  rate.  The  BFA 
rate  is  the  percentage  of  attempted  sorties 
that  fail  to  become  airborne  because  of 
failures  discovered  by  the  aircrew  before 
takeoff.  The  IFA  rate  is  the  percentage  of 
sorties  which  become  airborne  that  subse¬ 
quently  fail  to  complete  the  defined  mission 
because  of  a  failure  discovered  during  flight. 
The  total  abort  rate  is  the  sum  of  BFA  and 
IFA  rates. 

(2)  Captive-Carry  Reliability  (CCR).  The 
probability  a  missile  or  munition  will  remain 
failure  free  while  loaded  and  carried  on  the 
host  aircraft. 

(3)  Dormant  Reliability  fDR).  The  proba¬ 
bility  an  item  will  remain  failure  free  for  a 
specified  period  of  time  in  an  nonoperating 
mode  under  stated  environmental  conditions. 

(4)  Launch  and  Flight  Reliability  (LFR). 
The  probability  a  munition,  available  for 
commitment  to  the  launch  sequence,  will 


respond  to  a  valid  launch  command  and 
successfully  complete  the  launch  and  flight 
’.vi th  delivery  of  a  given  warhead  within 
accuracy  requirements.  This  term  includes 
a  combination  of  operational  suitability  and 
effectiveness  data. 

(5)  Mean  Mission  Duration  (MMD).  The 
average  interval  of  time  over  which  a  system 
is  expected  to  operate  without  mission  fail¬ 
ure. 

(6)  Mean  Time  Between  Critical  Failure 
(MTBCF).  The  average  time  between  failure 
or  unacceptable  degradation  of  essential 
system  functions.  Essential  system  functions 
are  those  which  must  be  operational  if  the 
mission  is  to  succeed.  MTBCF  should  con¬ 
sider  software  as  well  as  hardware  failures. 
Separate  calculations  should  be  made  for 
software,  hardware,  and  composite  MTBCF. 
The  following  formula  applies: 

MTBCF  =  total  operating  hours 

number  of  critical  events 

MTBCF  is  a  major  parameter  of  weapon 
system  reliability. 

(7)  Mean  Time  Between  Downing  Events 
(MTBDE).  A  measure  related  to  availability: 
the  total  number  of  system  life  units  divided 
by  the  total  number  of  events  in  which  the 
system  becomes  unavailable  to  initiate  its 
missions  during  a  stated  period  of  time. 

(8)  Weapon  System  Reliability  (WSR). 
The  probability  a  system  will  complete  a 
specified  mission,  given  the  system  was 
initially  capable  of  performing  that  mission. 
WSR  is  a  measure  of  system  reliability  as  it 
affects  the  mission  and  includes  the  effects 
of  both  hardware  and  software  critical  fail¬ 
ures  (faults).  WSR  excludes  operational 
effectiveness  factors  such  as  probability  of 
kill,  circular  error  probable,  fault  detection 
probability,  and  other  measures  of  capability. 
WSR  calculations  are  oriented  toward  specific 
mission  scenarios  and  require  the  identifica¬ 
tion  of  mission  length  by  mission  phase, 
mission  essential  systems/subsystems  by 
mission  phase,  and  the  life  units  of  the 
systems/subsystems  during  each  mission 
phase. 

b.  Logistics-Related  Reliability.  Relia¬ 
bility  measures  selected  to  account  for  or 
address  all  incidents  that  require  a  response 
from  the  logistics  system.  Logistics-related 
reliability  may  be  further  subdivided  into 
maintenance-related  reliability  and  supply- 
related  reliability. 

(1)  Maintenance-Related  Reliability. 
Mean  time  between  maintenance  (MTBM)  is 
the  primary  measure  of  logistics  reliability. 
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To  obtain  additional  insight  into  the  system 
under  test,  a  maintenance  action  can  be 
categorized  as  caused  by  inherent  malfunc¬ 
tion,  induced  malfunction,  or  no  defect. 
Separate  computations  of  MTBM  (inherent), 
MTBM  (induced),  and  MTBM  (no  defect)  may 
be  made  in  addition  to  total  MTBM.  MTBM 
is  the  total  time  in  hours  (for  example, 
operating,  flight,  or  possessed)  divided  by  the 
total  number  of  maintenance  (base-level  on- 
equipment)  events  for  a  specified  period  of 
time.  (See  AFLCR  66-15  for  a  discussion  of 
the  relationships  between  types  of  main¬ 
tenance  events  and  how-malfunction/action- 
taken-codes. )  Expressing  the  time  interval 
in  terms  of  flying  or  operating  hours  is 
useful  for  systems  that  fail  in  use.  Mean 
sorties  between  maintenance  (MSBM)  is  more 
meaningful  for  systems  whose  failure  is 
based  on  operating  cycles,  e.g.,  startup  fail¬ 
ures  in  a  computer  system  or  landing  gear 
malfunctions.  Although  the  number  of  sor¬ 
ties  is  not  exactly  equivalent  to  the  oper¬ 
ating  cycles,  the  use  of  MSBM  will  lend 
insight  to  failures  of  a  cyclic  nature.  It  is 
important  MTBM  not  be  confused  with  the 
contractual  term  MTBF.  MTBF  is  normally 
computed  using  only  inherent  failures  or  a 
subset  of  inherent  failures.  The  AFLC  data 
system  which  tracks  historical  MTBM  per¬ 
formance  is  the  D056,  Product  Performance 
System. 

(2)  Supply-Related  Reliability: 

(a)  Mean  Time  Between  Demand 
(MTBD).  A  measure  of  system  reliability 
related  to  demand  for  logistics  support.  It  is 
the  total  number  of  system  life  units  divided 
by  the  total  number  of  item  demands  on  the 
supply  system  during  a  stated  period  of  time. 
AFLCR  57-4,  Recoverable  Consumption  Item 
Requirements  System,  defines  the  demands. 
The  AFLC  data  system  which  tracks  histori¬ 
cal  MTBD  performance  is  the  D041,  Re¬ 
coverable  Consumption  Item  Requirements 
System. 

(b)  Mean  Time  Between  Removal 
(MTBR).  MTBR  is  equal  to  the  total  number 
of  system  life  units  divided  by  the  total 
number  of  items  removed  from  that  system 
during  a  stated  period  of  time.  This  term 
excludes  removals  performed  to  facilitate 
other  maintenance  and  removals  to  accom¬ 
plish  time  compliance  technical  orders. 

10-6.  System  Reliability  Models: 

a.  System  reliability  models  visually  and 
mathematically  describe  the  relationship 
between  the  reliability  of  system  components 
and  the  resulting  system  reliability.  A 
reliability  block  diagram  or  structural  model 
provides  a  visual  representation,  whereas  a 


mathematical  model  provides  an  analytical 
tool  to  calculate  quantitative  reliability 
values. 

b.  The  following  notation  is  used  in  the 
discussion  of  reliability  models. 

Rs  =  reliability  of  the  system 

R.  =  reliability  of  the  ith  subsystem 

Qs  =  1  -  Ra  =  unreliability  of  the  system 

=  1  -  R;  =  unreliability  of  the  ith  sub¬ 
system 

TT  =  "product  of'  (Note:  This  operator  is 
used  in  the  same  fashion  as  I  for  summa¬ 
tion,  but  it  indicates  multiplication  rather 
than  addition.) 

c.  The  following  discussion  assumes  all 
subsystems  function  independently  from  one 
another;  that  is,  failures  of  different  sub¬ 
systems  are  statistically  independent  of  each 
other.  For  some  systems,  this  represents  a 
realistic  assumption;  for  others,  it  does  not. 
The  reliability  analysis  for  dependent  subsys¬ 
tems  is  significantly  more  complex.  Inde¬ 
pendent  operation,  practically  speaking, 
means  a  failure  of  one  system  will  not  cause 
a  change  in  the  failure  characteristics  of 
other  subsystems. 

(1)  Series  Model.  When  a  group  of 
components  or  subsystems  is  such  that  all 
must  function  properly  for  the  system  to 
succeed,  they  are  said  to  be  in  series.  A 
system  consisting  of  a  series  arrangement 
in  subsystems  is  illustrated  in  the  following 
block  diagram: 


The  mathematical  model  is: 

^8  =  =  F  Rj 

i=l 

(2)  Redundant  Models.  The  mission 
reliability  of  a  system  containing  independent 
subsystems  can  usually  be  increased  by 
adding  redundant  subsystems.  The  incor¬ 
poration  of  redundancy  into  a  system  design 
and  the  subsequent  analysis  and  assessment 
of  that  design  is  a  complex  task  and  will  not 
be  addressed  here  in  detail  (see  RADC’s 
reliability  engineers  toolkit  for  a  more  com¬ 
plete  discussion).  Our  discussion  will  con¬ 
sider  only  simple  active  redundancy.  In  this 
type  of  redundancy,  all  the  operable  subsys¬ 
tems  are  functioning,  but  only  one  is  needed 
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for  satisfactory  performance.  There  are  no 
standby  subsystems,  and  no  repair  is  per¬ 
mitted  during  the  mission.  Such  a  system 
is  illustrated  in  block  diagram  form  as: 


R5  =  R1RnR3  [l-(  1-R4;(  1-R3i(  1-Rs;j 

(4)  Functional  Models: 

(a)  The  redundant  and  mixed  models 
series  mentioned  above  are  hardware-oriented 
in  that  they  display  hardware  capabilities.  In 
some  cases,  it  is  desirable  to  model  a  system 
from  a  functional  standpoint.  As  an  ex¬ 
ample,  the  functional  reliability  block  dia¬ 
gram  for  a  fighter  aircraft  is  shown  below: 


The  mathematical  model  is: 

Q,  =  QiQj.-Qn  =  F  Qj  =  ¥  (l-R.) 

i=l  i=l 


R.  -  i  -  Q.  -  i  * 


ir  (1  -  R|) 
i=l 


(3)  Mixed  Models: 

(a)  A  system  configuration  that  is  often 
encountered  is  one  in  which  subsystems  are 
in  series,  but  redundancy  (active)  is  applied 
to  a  certain  critical  subsystem.  A  typical 
block  diagram  follows: 


(b)  This  model  (or  any  mixed  model)  is 
characterized  by  working  from  low  to  high 
levels  of  assembly.  In  this  case,  the  equation 
for  simple  active  redundancy  which  requires 
at  least  one  of  components  4,  5,  or  6  to 
function  can  be  applied: 

^4,5, 6  =  H1-R4XTR5XTR6) 

(c)  T!i?  redundant  configuration  of  4,  5, 
and  6  can  be  then  represented  by  a  single 
block  on  the  diagram. 


Now  the  equation  for  a  series  model  can  be 
applied: 


(b)  Note  this  concept  addresses  mission- 
essential  functions  but  in  no  way  implies 
how  these  functions  will  be  accomplished. 
Generally,  the  functional  model  is  helpful  in 
the  program  formulation  stages  of  a  program 
when  specific  hardware  information  is  not 
necessary  and  frequently  not  desired.  This 
type  of  model  can  provide  a  useful  transition 
from  operational  requirement  to  engineering 
specification. 

10-7.  Statistical  Test  Design: 
a.  Test  Design: 

(1)  One  of  the  underlying  purposes  for 
reliability  testing  is  to  determine  the  aging 
characteristics  of  the  item  so  inferences  can 
be  made  as  to  how  the  item  might  perform 
during  the  execution  of  a  mission.  This  is 
usually  done  by  collecting  data  during  OT&E 
and  using  it  to  estimate  reliability  param¬ 
eters.  By  evaluating  the  parameters  on  a 
few  test  items,  inferences  can  be  drawn  on 
how  a  fleet  of  items  will,  on  the  average, 
respond  to  required  operations. 

(2)  There  is  a  wide  range  of  test  struc¬ 
tures  or  procedures  used  to  learn  about  item 
or  system  failures  and  their  frequency  of 
occurrence.  A  statistical  test  design  results 
from  the  specification  of  the  conditions  to  be 
used,  the  definition  and  categorization  of  test 
events  (or  failures),  the  items  of  data  to  be 
collected,  and  the  methodology  to  be  used  in 
evaluating  the  resulting  data.  If  the  reliabil¬ 
ity  analyst  has  little  or  no  influence  on  the 
specification  of  the  test  conditions,  the  ana¬ 
lyst  may  be  forced  to  adapt  analysis  tech¬ 
niques  resulting  in  less  rigorous  evaluation 
methods.  Although  there  are  many  types  of 
reliability  testing  methods,  in  general,  relia¬ 
bility  testing  before  and/or  during  OT&E  is 
concerned  with  developmental  or  acceptance 
type  testing.  These  tests  are  used  for  esti¬ 
mating  reliability  parameters  and  deciding 
whether  the  parameters  have  reached  an 
acceptable  level  at  a  certain  degree  of  confi- 
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dence  at  the  particular  stage  of  system 
acquisition,  i.e.,  meets  threshold,  goal,  or 
contractual  values. 

b.  Statistical  Test  Design  Approach, 
Most  test  designs  involve  the  determination 
or  estimation  of  the  parameters  of  an  under¬ 
lying  time  to  failure  distribution.  The  most 
common  time-to-failure  distribution  used  in 
reliability  testing  is  the  exponential  distribu¬ 
tion,  although  some  cases  are  better  repre¬ 
sented  by  other  distributions,  such  as  the 
Weibull  or  Normal.  In  OT&E,  especially  in 
the  preliminary  or  planning  phases  of  relia¬ 
bility  test  design,  an  exponential  distribution 
is  frequently  assumed  for  analysis  purposes. 
If  sufficient  data  are  collected  during  the 
OT&E,  and  if  time  and  resources  permit, 
further  analysis  of  the  underlying  time  to 
failure  distribution  should  be  made. 

c.  Test  Planning  Considerations: 

(1)  Determining  the  test  item  sample  size 
and  test  time  must  be  done  far  enough  in 
advance  of  the  testing  to  include  the  re¬ 
sources  in  the  test  program  outline  and 
ultimately  to  allow  the  SPO  to  identify  the 
funds  required  in  the  budget  cycle.  Inade¬ 
quate  sample  size  can  easily  negate  the 
validity  of  reliability  test  results. 

(2)  Hypothesis  testing  can  be  used  once 
the  time  to  failure  distribution  has  been 
assumed  or  estimated.  The  process  involves 
the  determination  of  a  null  hypothesis  to  be 
tested  and  an  alternative  hypothesis  which 
will  be  assumed  true  if  the  null  hypothesis 
is  rejected.  The  sample  size  or  test  time 
determines  statistically  the  risks  associated 
with  such  testing.  There  is  a  risk  of  re¬ 
jecting  the  null  hypothesis  when  it  is  in  fact 
true  and  a  risk  of  accepting  the  null  hypoth¬ 
esis  when  in  fact  the  alternative  hypothesis 
is  true.  For  OT&E,  however,  the  number  of 
test  assets  as  well  as  the  length  of  the  test 
period  is  likely  to  be  constrained  because  of 
cost  considerations.  Tradeoffs  between  the 
risk  levels  can  be  made,  but  the  required 
sample  size  becomes  larger  for  a  higher 
degree  of  certainty  of  making  the  correct 
decision.  The  test  design  should  state  the 
risk  levels  used  for  the  hypothesis  test. 

(3)  Early  communication  of  reliability 
test  requirements  must  be  emphasized  be¬ 
tween  AFOTEC,  the  SPO,  and  the  using 
command.  These  communications  should 
result  in  a  specification  of  contractual  relia¬ 
bility  consistent  with  operational  reliability 
levels  and  an  acceptable  risk  level  to  the 
accept  or  reject  decision.  In  many  cases,  the 
reliability  and  risk  specifications  of  the  SPO 
or  using  command  would  require  an  unrea¬ 
sonable  total  test  time  and/or  number  of  test 
assets.  When  this  situation  occurs,  the 
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approving  authority  (Air  Staff  or  OSD)  must 
reduce  the  required  confidence  level  and/or 
required  reliability  level  or  increase  the  test 
funding  for  more  time  and/or  assets.  In  any 
case,  the  approving  authority  should  have  a 
concrete  understanding  of  what  the  reliability 
testing  can  determine,  given  various  tradeoffs 
between  test  time,  reliability  specifications, 
and  levels  of  confidence  in  reliability  predic¬ 
tions.  Tradeoffs  between  demonstrated 
reliability  and  total  test  time  required  at 
various  confidence  levels  can  be  plotted. 
Many  examples  of  test  design  specification  as 
well  as  a  much  more  detailed  discussion  of 
hypothesis  testing,  sample  size,  level  of 
confidence  calculations,  and  parameter  esti¬ 
mation  are  provided  in  the  DOD  Primer 
DOD  3235. 1-H,  Test  and  Evaluation  of  Sys¬ 
tem  Reliability,  Availability,  and  Maintaina¬ 
bility. 

(4)  The  selection  of  MOEs  for  reliability 
should  be  based  on  the  terms  found  in  AFP 
57-9  and  the  ORD. 

(5)  WSR  should  be  calculated  for  each 
type  of  mission  and  may  be  used  to  accom¬ 
plish  a  comprehensive  mission  effectiveness 
analysis.  For  complex  systems,  modeling  is 
required  to  estimate  WSR.  However,  for 
simple  systems,  WSR  is  often  calculated 
using  the  following  formula: 

WSR  =  e 

where 

-  failure  rate  =  1 

MTBCF 

t  =  estimated  mission  length 

e  =  the  base  of  the  natural  logarithm 
(2.71828) 

NOTE:  This  formula  assumes  a  constant 
failure  rate  (i.e.,  exponential  distribution).  If 
this  assumption  is  invalid,  other  probability 
distributions  will  be  used. 

(6)  Incoming  inspection  acceptance  rate 
is  an  MOE  peculiar  to  munition/missile 
programs.  All  munitions/missiles  require  an 
incoming  inspection  to  check  for  damage  and 
serviceability  when  they  are  first  received  at 
a  base.  The  inspection  may  vary  from  a 
visual  inspection  of  containers  to  a  full 
functional  checkout  on  a  test  set.  The  in¬ 
coming  inspection  acceptance  rate  is  the 
percent  of  munitions/missiles  that  pass  the 
incoming  inspection.  Incoming  inspection 
procedures  are  normally  established  and  well 
documented  by  the  time  a  test  program 
starts  because  of  the  inherent  hazards  as- 
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sociated  with  transporting  and  handling 
munitions. 

(7)  Dormant  reliability  is  also  peculiar 
to  munitions/missiles  programs.  These  sys¬ 
tems  usually  spend  the  majority  of  their  life 
in  storage.  The  current  trend  toward  "wood¬ 
en  rounds"  (systems  which  are  designed  to 
require  no  maintenance)  also  increases  the 
importance  of  dormant  reliability,  since  there 
will  be  no  capability  to  verify  system  status 
before  use.  Unfortunately,  dormant  reliabil¬ 
ity  is  extremely  difficult  to  measure  during 
IOT&E  because  of  the  limited  time  and  test 
assets  available.  The  primary  effort  during 
IOT&E  should  be  devoted  to  determining 
contractor  and  SPO  design  efforts  and  test 
programs  to  ensure  reliability  in  the  dormant 
state.  For  more  discussion  of  this  topic,  see 
attachment  2. 

(8)  Data  requirements  for  reliability 
consist  of  the  following  elements  and  sources: 

(a)  Elements.  The  data  elements  are 
typically  the  same  for  IOT&E  and  FOT&E. 

1.  Mission  length. 

2.  Number  of  critical  failures. 

3.  Number  of  maintenance  events. 

4.  Mission  essential  subsystem  list. 

5.  Work  unit  codes. 

6.  Number  of  sorties. 

7.  Operating  hours. 

(b)  Sources.  Data  sources  will  depend 
on  the  testing  environment. 

1.  During  IOT&E,  SEDS  may  be  used 
to  collect  data  elements.  A  joint  reliability 
and  maintainability  evaluation  team 
(JRMET)  is  normally  formed  and  chaired  by 
the  system  program  office  (SPO)  to  categorize 
maintenance  events  and  identify  critical 
failures.  SEDS  is  structured  to  allow  OT&E 
and  DT&E  interpretations  of  maintenance 
events  to  be  reflected  in  the  data  base.  If 
SEDS  is  unavailable  or  inappropriate,  a 
tailored  data  collection  system  must  be 
developed. 

2.  During  FOT&E,  the  MDC  system 
is  typically  used  to  collect  maintenance  data. 
MDC  documentation,  along  with  crew  debrief¬ 
ing  forms,  is  then  reviewed  to  identify  criti¬ 
cal  failures.  The  number  of  flying  hours/ 
sorties  may  be  extracted  from  MMICS.  The 
AFOTEC  OMNIVORE  data  system  may  be 
used  to  maintain  the  data  base. 

d.  Analysis  of  Test  Results.  Whether 
or  not  a  statistically  significant  reliability 
test  plan  was  formally  structured  before 
undergoing  the  actual  OT&E,  data  concerning 
the  frequency  and  type  of  system  failures 
during  the  test  period  will  have  been  gener¬ 
ated.  If  a  reliability  test  plan  was  fully 
structured  and  adhered  to  during  the  test, 
the  computation  of  whether  the  system  meets 


or  exceeds  required  reliability  values  is 
straightforward.  In  many  cases,  insufficient 
test  time  and/or  assets  under  test  will  limit 
what  can  be  confidently  said  about  the 
reliability.  In  those  cases,  simply  reporting 
the  observed  reliability  under  various  defini¬ 
tions,  along  with  some  estimation  of  con¬ 
fidence  limits  around  the  mean,  may  be  all 
that  can  be  accomplished  analytically.  As 
an  extreme  example,  if  one  were  to  try  to 
estimate  the  dormant  reliability  of  a  missile 
for  which  27,108  hours  of  ground  inactive 
test  time  was  accumulated  and  for  which  one 
failure  was  observed,  an  80-percent  confi¬ 
dence  interval  for  reliability  would  range 
from  6,969  to  257,288  hours.  When  little  or 
no  meaningful  reliability  data  are  available, 
the  emphasis  of  the  reliability  assessment 
should  shift  from  quantitative  to  qualitative. 

10-8.  The  Need  for  Projection  Capabil¬ 
ity  During  OT&E  -  Background: 

a.  The  initial  design  for  a  complex  sys¬ 
tem  will  invariably  have  significant  relia¬ 
bility  deficiencies  that  could  not  be  foreseen 
in  the  early  stages  of  the  design  effort. 
These  immature  designs  may  be  subjected  to 
a  structured  test  program  to  identify  prob¬ 
lems  so  improvements  can  be  made.  Any 
improvement  in  system  reliability,  i.e.,  relia¬ 
bility  growth,  will  depend  on  the  number  and 
effectiveness  of  system  design  improvements/ 
changes.  An  ultimate  goal  of  a  reliability 
growth  program  is  to  meet  or  exceed  system 
reliability  requirements  as  outlined  by  the 
user. 

b.  Point  or  interval  estimates  of  relia¬ 
bility,  calculated  from  current  and  past 
observations,  can  provide  an  overall  measure 
of  reliability  performance  and  can  be  used  as 
a  management  indicator.  However,  during 
operational  test  and  evaluation  of  systems  in 
the  acquisition  process,  measured  reliability 
characteristics,  in  and  of  themselves,  do  not 
provide  reasonable  indications  of  expected 
future  system  reliability.  Expected  future 
reliability  predictions  must  be  made  in  the 
context  of  expected  future  system  characteris¬ 
tics  (design  and  use).  The  predictions  must 
be  continuously  refined  as  the  system  ma¬ 
tures  and  test  data  are  replaced  by  more 
operationally  representative  data. 

c.  Three  types  of  reliability  estimates  can 
be  defined: 

(1)  Demonstrated  or  observed  reliability 
estimates  are  those  obtained  directly  from 
the  test  program. 

(2)  Current  (instantaneous  reliability 
estimates  are  those  (vertical)  adjustments  to 
demonstrated  reliability  by  a  growth  factor. 
The  growth  factor  accounts  for  system  design 
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improvements  to  correct  identified  problems. 

(3)  Projected  reliability  estimates  are 
those  (horizontal,  or  horizontal  and  vertical) 
adjustments  which  extend  the  current  relia¬ 
bility  estimate  forward  in  time.  These  esti¬ 
mates  assume  a  continuous  process  of  system 
design  improvements  to  correct  identified 
problems  and  problems  remaining  in  the 
system.  (Reference  MIL-HDBK  189,  Reliabil¬ 
ity  Growth  Management.) 

10-9.  Application  of  Projections.  Confu¬ 
sion  has  arisen  regarding  AFOTEC’s  applica¬ 
tion  of  reliability  growth  theory  to  the  test 
and  evaluation  process.  Some  people  have 
been  under  the  impression  since  OT&E 
thresholds  are  stated  in  terms  of  user  re¬ 
quirements  for  a  mature  (IOC  plus  2  years 
in  most  cases)  system,  AFOTEC  compares  re¬ 
sults  demonstrated  during  OT&E  to  mature 
system  requirements  when  assigning  a  rat¬ 
ing.  Actually,  reliability  values  measured 
during  OT&E  are  projected  to  maturity  using 
accepted  reliability  growth  theory  before 
being  compared  with  user  requirements  for 
a  mature  system.  Thus,  test  ratings  are 
based  on  projections  of  a  system’s  expected 
reliability  rather  than  the  actual  measured 
OT&E  experience.  Note:  Accepted  practice 
is  to  report  both  the  reliability  observed 
during  OT&E  and  the  projected  reliability. 

10-10.  Reliability  Growth  and  Tests: 

a.  Reliability  Improvement.  As  high¬ 
lighted  in  MIL-STD  785,  testing  does  not 
improve  reliability.  Only  corrective  actions 
that  prevent  the  recurrence  of  failures  in  the 
operational  inventory  actually  improve  relia¬ 
bility.  In  most  cases,  this  will  require  addi¬ 
tional  funding  for  equipment  or  design 
changes  and  a  retest  to  evaluate  the  changes. 

b.  Reliability  Growth.  DOD  policy 
states  reliability  growth  is  required  during 
full-scale  development,  concurrent  develop¬ 
ment  and  production  (where  concurrence  is 
approved),  and  during  initial  deployment. 
Predicted  reliability  is  stated  as  a  series  of 
intermediate  milestones,  with  associated  goals 
and  thresholds,  for  each  specified  parameter 
for  each  of  those  phases.  A  period  of  testing 
is  scheduled  in  conjunction  with  each  inter¬ 
mediate  milestone.  Approved  reliability 
growth  requirements  are  assessed  and  en¬ 
forced  at  decision  milestones. 

c.  System  Reliability.  This  testing 
philosophy  utilizes  the  test-analyze-fix-test 
(TAFT)  procedure  as  the  basic  catalyst  in 
achieving  system  reliability  growth.  The  goal 
of  a  reliability  growth  program,  and,  indeed, 
the  entire  test  program,  is  to  increase  system 
reliability  to  stated  requirement  levels  by 


eliminating,  or  reducing,  a  sufficient  number 
of  inherent  system  failure  modes. 

d.  Growth  Program.  A  successful 
system  reliability  growth  program  depends  on 
several  factors.  First,  an  accurate  determina¬ 
tion  must  be  made  of  the  current  system's 
reliability.  Second,  a  test  program  must  be 
planned  which  subjects  the  system  to  test 
exposure  and  stress  levels  adequate  to  un¬ 
cover  inherent  failure  modes  and  to  verify 
design  modifications.  Third,  the  program 
manager  must  address  the  availability  of  test 
schedule  and  resources  required  to  support 
the  TAFT  procedures. 

e.  Growth  Rate.  To  adequately  control 
factors  inherent  in  the  reliability  growth 
throughout  the  test  program.  This  is  accom¬ 
plished  by  periodically  assessing  system 
reliability  (e.g.,  at  the  end  of  every  test 
phase)  and  comparing  the  current  reliability 
to  the  planned  level  of  achievement  for  that 
point  in  time.  These  assessments  provide 
the  necessary  data  and  visibility  to  support 
necessary  corrective  management  initiatives. 
The  rate  at  which  reliability  growth  occurs 
(i.e.,  the  growth  rate!  provides  the  primary 
process-oriented  measure  for  assessing  a 
system’s  potential  for  achieving  mature 
system  reliability. 

£  Types  of  Development  and  Produc¬ 
tion  Reliability  Testing.  Reliability  test 
programs  serve  three  objectives:  (1)  disclose 
deficiencies  in  item  design,  material,  and 
workmanship;  (2)  provide  measured  reliability 
data  as  input  for  estimates  of  operational 
readiness,  maintenance  manpower  cost, 
logistics  support  cost,  etc.;  and  (3)  determine 
compliance  with  quantitative  reliability 
requirements.  Four  types  of  reliability  tests 
of  interest  to  T&E  personnel  are  (1)  en¬ 
vironmental  stress  screening  (ESS),  (2)  re¬ 
liability  development/growth  testing  <RDGT), 

(3)  reliability  qualification  tests  (RQT),  and 

(4)  production  reliability  acceptance  tests 
(PRAT).  The  ESS  and  RDGT  are  classified 
as  engineering  tests;  RQT  and  PRAT  are 
classified  as  accounting  tests. 

( 1)  Environmental  Stress  Screening  ( ESS). 
ESS  is  a  test,  or  a  series  of  tests,  specifically 
designed  to  disclose  weak  parts  and  work¬ 
manship  defects  for  correction.  It  should  be 
applied  to  parts,  components,  subassemblies, 
assemblies,  or  equipment  ( as  appropriate  and 
cost-effective)  to  remove  defects  which  would 
otherwise  cause  failures  during  higher  level 
testing  or  early  field  service.  The  test  condi¬ 
tions  and  procedures  should  be  designed  to 
stimulate  failures  typical  of  early  field  serv¬ 
ice,  rather  than  to  provide  precise  simula¬ 
tion  of  the  operational  life  profile.  These 
tests  should  be  considered  an  early  portion 
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of  reliability  development/growth  testing. 
They  must  be  conducted  early  in  development 
to  ensure  time  and  resources  are  available  to 
correct  the  deficiencies  they  disclose  and  to 
verify  the  corrections. 

(2)  Reliability  Development/Growth  Test¬ 
ing; 

(a)  RDGT  is  a  planned,  prequalifi¬ 
cation,  test-analyze-fix  process  in  which 
equipments  are  tested  under  actual,  simu¬ 
lated,  or  accelerated  environments  to  disclose 
design  deficiencies  and  defects.  RDGT  pro¬ 
vides  a  basis  for  early  incorporation  of  correc¬ 
tive  actions  and  verification  of  their  effective¬ 
ness,  thereby  promoting  reliability  growth. 

(b)  Predicted  reliability  growth  must 
differentiate  between  the  apparent  growth 
achieved  by  screening  weak  parts  and  work¬ 
manship  defects  out  of  the  test  items  and  the 
step-function  growth  achieved  by  design 
corrections.  The  apparent  growth  resulting 
from  ESS  does  not  transfer  from  prototypes 
to  production  units;  instead  it  repeats  in 
every  individual  item  of  equipment.  The 
step-function  growth  does  transfer  to  produc¬ 
tion  units  that  incorporate  effective  design 
corrections.  Therefore,  RDGT  plans  should 
include  a  series  of  test  periods  and  each  of 
the  test  periods  should  be  followed  by  a  'fix" 
period  (step-function  growth).  RDGT  should 
be  conducted  using  one  or  two  of  the  first 
full-scale  engineering  development  items 
available. 

(3)  Reliability  Qualification  Test.  RQT 
is  intended  to  provide  the  government  rea¬ 
sonable  assurance  that  minimum  acceptable 
reliability  requirements  have  been  met  before 
items  are  committed  to  production.  RQT 
must  be  operationally  realistic,  with  prede¬ 
fined  criteria  to  limit  the  risk  the  item  may 
have  a  true  reliability  less  than  the  mini¬ 
mum  acceptable  reliability.  RQT  is  required 
for  items  newly  designed,  have  undergone 
major  modification,  and  have  not  met  their 
allocated  reliability  requirements. 

(4)  Production  Reliability  Test  (PRAT). 
PRAT  is  intended  to  simulate  in-service 
evaluation  of  the  delivered  item  or  production 
lot.  It  must  be  operationally  realistic  and 
can  consist  of  a  normal  test,  an  overload  test, 
and/or  a  mission  profile  cycling  test  that 
duplicates  or  approximates  the  conditions 
expected  in  service. 

g.  Test  Realism.  A  test  is  realistic  to 
the  degree  that  test  conditions  and  proce¬ 
dures  simulate  the  operational  life/mission/ 
environmental  profile  of  a  production  item. 
Realistic  testing  can  disclose  deficiencies  and 
defects  that  otherwise  would  be  discovered 
only  after  an  item  is  deployed.  Test  realism 
must  be  a  primary  consideration  in  every 


reliability  test.  Establishment  of  realistic 
test  conditions  and  procedures  requires  a 
knowledge  of  the  life  profile  from  factory  to 
final  expenditure/retirement. 

h.  Integrated  Testing.  It  is  DOD  policy 
that  reliability  and  environmental  stress  tests 
can  be  combined  as  far  as  practical.  For 
example,  mechanical,  hydraulic,  pneumatic, 
and  electrical  equipment  are  usually  sub¬ 
jected  to  three  qualification  tests:  perform¬ 
ance,  environmental,  and  endurance  or  dura¬ 
bility.  The  integration  of  these  separate 
tests  into  a  more  comprehensive  reliability 
test  program  can  avoid  costly  duplication 
and  ensure  deficiencies  are  not  overlooked  as 
they  often  are  in  the  fragmented  approach. 

i.  Combined  Environmental  Reliability 
Test  (CERT): 

(1)  Studies  have  shown  approximately 
one-half  of  field  failures  are  environment 
induced.  These  studies  also  have  shown 
MIL-STD  environmentally  based  tests  used 
for  equipment  qualification  may  not  give 
realistic  assessments  of  an  equipment’s 
reliability  in  its  operational  environment. 
Further  analysis  of  environmental  data  shows 
the  environmental  conditions  of  temperature, 
humidity,  altitude,  and  vibration  do  not 
generally  remain  constant  throughout  an 
aircraft’s  or  missile’s  mission.  Therefore,  the 
relationship  between  aircraft  (or  missile) 
flight  conditions  and  the  operational  environ¬ 
ment  of  the  aircraft  (or  missile)  and  its 
avionics  are  of  primary  concern.  Environ¬ 
mental  configurations  in  a  test  scenario 
should  be  presented  in  a  time  sequence 
similar  to  the  environmental  conditions 
experienced  by  an  operational  aircraft  or 
missile.  This  concept  of  providing  laboratory 
test  conditions  representative  of  the  field  is 
called  combined  environmental  reliability  test 
(CERT). 

(2)  CERT  is  a  series  of  laboratory  tests 
that  attempt  to  duplicate  flight  environment¬ 
al  conditions  by  using  mission  profile  tests. 
Combinations  of  environmental  conditions 
characteristic  of  those  that  occur  during  a 
typical  aircraft  mission  are  put  together  in 
a  test  sequence.  This  involves  the  simul¬ 
taneous  simulation  in  a  test  chamber  of  the 
temperature,  humidity,  altitude,  vibration, 
and  cooling  air  mass  flow  into  a  time  history 
sequence  as  if  the  test  item  were  in  actual 
flight.  Thus,  the  behavior  of  the  equipment 
in  this  laboratory  test  should  closely  ap¬ 
proach  its  field  performance.  A  high  degree 
of  correlation  exists  between  CERT  and  field 
experience  in  terms  of  failure  rates  and 
modes.  CERT,  used  in  conjunction  with  a 
test-analyze-fix  growth  approach,  can  be  very 
effective  for  early  identification  of  defiden- 
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cies. 

(3)  Although  CERT  is  a  reliability  test 
used  by  the  developer  during  the  system’s 
acquisition,  AFOTEC  should  make  maximum 
use  of  CERT-generated  data  for  the  system's 
evaluation.  CERT,  if  applied  by  the  devel¬ 
oper  during  the  system’s  acquisition,  may 
provide  the  OT&E  community  with  a  cost- 
effective  tool  for  identifying  deficiencies  and 
verifying  their  correction, 
j.  Reliability  Growth  Concepts: 

(1)  Idealized  Growth: 

(a)  For  a  system  under  development, 
reliability  generally  increases  rapidly  early  on 
and  at  a  much  slower  rate  toward  the  end  of 
development.  It  is  useful  at  the  beginning 
of  a  development  program  to  depict  the 
growth  in  reliability  as  a  smooth  curve  which 
rises  at  slower  and  slower  rates  as  time 
progresses.  This  curve,  known  as  the  ideal¬ 
ized  growth  curve,  does  not  necessarily 
convey  how  the  reliability  will  actually  grow 
during  development.  Its  purpose  is  to  pre¬ 
sent  a  preliminary  view  of  how  a  program 
should  be  progressing  in  order  to  realize  the 
final  requirements. 

(b)  The  development  testing  program 
will  usually  consist  of  several  major  test 
phases.  If  we  divide  the  development  testing 
program  into  its  major  phases  and  join  by  a 
smooth  curve  the  proposed  reliability  values 
for  the  system  at  the  end  of  these  test 
phases,  the  resulting  curve  represents  the 
overall  pattern  for  reliability  growth  (see 
figure  10-1).  This  idealized  curve  is  very 
useful  in  quantifying  the  overall  development 
effort  and  serves  as  a  significant  tool  in  the 
planning  of  reliability  growth.  One  model 
for  the  idealized  growth  curve  is  the  Duane 
Growth  Model.  The  T&E  Primer,  DOD 
3235. 1-H,  contains  an  example  of  how  to 
construct  an  idealized  growth  curve  using  the 
Duane  Growth  Model. 

(2)  Planned  Growth: 

(a)  Reliability  growth  planning  is  done 
early  in  the  development  program,  before 
hard  reliability  data  are  obtained,  and  is 
typically  a  joint  effort  between  the  program 
manager  and  the  contractor.  AFOTEC 
should  participate  in  this  effort  to  ensure  we 
have  the  necessary  interim  measurement 
points  for  test.  The  objective  of  growth 
planning  is  to  determine  the  number  and 
length  of  distinct  test  phases,  whether  design 
modifications  will  be  incorporated  during  or 
between  distinct  test  phases,  and  the  in¬ 
creases  in  reliability  necessary  to  ensure  the 
achieved  reliability  remains  within  sight  of 
the  idealized  growth  values. 

(b)  The  planned  growth  curve  displays, 
in  graphic  terms,  how  the  producer  plans  by 


stages  to  achieve  the  required  reliability. 
The  curve  is  divided  into  portions  which 
represent  the  different  test  phases  and  de¬ 
picts  increases  in  reliability  resulting  from 
design  improvements.  The  idealized  curve 
serves  as  a  guide  for  the  preparation  of  the 
planned  curve. 

(c)  As  mentioned  earlier,  the  planned 
growth  curve  should  display  how  reliability 
is  expected  to  grow,  usually  as  a  result  of 
incorporating  design  modifications  or  changes 
to  the  manufacturing  process.  These  modifi¬ 
cations  may  be  incorporated  during  the  test 
phase,  resulting  in  a  smooth  gradual  im¬ 
provement  in  reliability,  or  at  the  end  of  the 
test  phase,  resulting  in  a  jump  in  reliability 
from  the  end  of  one  test  phase  to  the  begin¬ 
ning  of  the  subsequent  test  phase. 

(d)  Figure  10-1  presents  a  planned 
growth  curve  which  illustrates  the  effect  on 
reliability  of  design  improvements  incorpor¬ 
ated  during,  and  at  the  completion  of,  the 
various  test  phases.  Delayed  fixes  are  incor¬ 
porated  after  each  of  the  first  three  test 
phases.  Fixes  are  incorporated  during  all  of 
test  phase  2  and  early  in  test  phase  3. 
Fixes  are  incorporated  during  the  final  test 
phase  and  the  time  between  failures  grow  to 
the  required  specified  value.  It  is  not  a  good 
practice  to  allow  for  a  jump  in  reliability  at 
the  end  of  the  final  test  phase  even  though 
fixes  may  be  incorporated,  since  there  is  no 
test  time  available  to  determine  the  impact 
of  these  fixes. 

k.  Caution  Regarding  Reliability 
Growth: 

(1)  It  cannot  be  overemphasized  that 
testing,  in  and  of  itself,  does  not  cause 
reliability  growth.  The  critical  element  of  a 
growth  program  is  a  comprehensive  plan  to 
analyze  failures  and  implement  design  modi¬ 
fications  to  eliminate  or  reduce  those  failures. 
Good  planning  and  adequate  funding  are  the 
necessary  catalysts  which  bring  about  relia¬ 
bility  growth. 

(2)  Operational  testers  must  guard 
against  estimating  mature  system  reliability 
by  using  only  the  mathematical  models  as  a 
basis  for  their  projections.  Considerations 
such  as  program  funding,  adequate  test  time 
and  resources,  and  a  commitment  on  the  part 
of  program  managers  to  implement  design 
modifications  are  critical  in  any  assessment 
of  system  reliability  at  maturity.  If  design 
modifications  stop  at  the  end  of  the  opera¬ 
tional  testing  phase,  reliability  will  not 
continue  to  improve.  Reliability  growth 
should  only  be  forecast  when  a  defined, 
funded  program  exists  to  allow  growth  to 
occur.  The  RMMP  (see  paragraph  10-2)  is 
the  source  document  for  describing  the  relia- 
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Figure  10-1.  Example  of  a  Planned  Growth  Curve  and 
Corresponding  Idealized  Curve 


bility  growth  program  for  the  system  under 
test. 

10-11.  Reliability  Evaluation: 

a.  Logistics  reliability  is  usually  evaluated 
by  analyzing  MTBMs  to  determine  their 
effect  on  the  ability  of  the  support  system  to 
respond  to  all  failures/maintenance  events 
under  the  defined  operational  and  support 
concepts  using  programmed  resources  and  to 
determine  the  effect  of  logistics  reliability  on 
the  system’s  capability  to  meet  mission 
requirements.  Additionally,  the  quantitative 
and  qualitative  data  are  analyzed  to  identify 
those  subsy stems/LRUs  demonstrating  relia¬ 
bility  below  the  required  or  expected  level, 
the  causes  for  poor  reliability,  and  those 
subsystems/LRUs  which  have  an  adverse 
impact  on  safety  and  man-hour  consumption. 
The  results  of  the  analysis  are  used  to  deter¬ 
mine  the  logistics  reliability  impact  on  the 
system’s  availability  and  to  assist  in  the 
evaluation  of  logistics  supportability. 

b.  Mission  reliability  data  are  analyzed 
to  assess  weapon  system  reliability  (WSR) 
and  to  identify  deficiencies  and  their  impact 
on  mission  success.  MTBCF  will  be  com¬ 
puted  for  subsystems/LRUs  and  aggregated 
at  the  system  level  for  calculating  a  probabil¬ 
ity  of  nonfailure  from  the  start  of  a  mission 
until  mission  termination.  Analysis  is  also 
performed  to  determine  which  subsystem  can 
be  improved  to  yield  the  largest  increase  in 
the  probability  of  a  failure-free  mission. 

10-12.  Lessons  Learned: 

a.  There  may  be  no  requirement  for  the 
contractor  to  provide  R&M  data  to  Air  Force 
agencies,  thereby  limiting  the  amount  of  data 
available  to  AFOTEC  test  personnel  during 
OT&E.  The  contract  data  requirements  list 


(CDRL)  should  be  reviewed  before  test  design 
activities  begin. 

b.  The  contractor  is  often  not  required  to 
use  an  Air  Force  data  collection  system 
during  test.  Problems  have  arisen  in  the 
past  because  only  relevant  failures  as  de¬ 
fined  by  the  contractor  were  included  in  the 
contractor’s  data  base.  In  addition,  the  data 
are  generally  not  auditable  and  the  quality 
of  the  data  may  be  questionable.  Logistics 
and  test  managers  should  attempt  to  require, 
via  the  contract,  the  use  of  an  Air  Force  data 
collection  system,  e.g.,  SEDS  (AFR  800-18). 

c.  A  common  agreement  on  failure  rele¬ 
vancy  between  the  DT&E  and  OT&E  person¬ 
nel  has  not  always  been  reached  during  a 
combined  test  because  a  JRMET  was  not 
established.  It  is  often  necessary  for 
AFOTEC  to  take  the  initiative  to  cause  the 
establishment  of  a  JRMET. 

d.  Test  results  can  be  distorted  if  equip¬ 
ment  is  not  operated  under  field  conditions. 
To  the  extent  possible,  all  systems  supporting 
an  article  undergoing  test  should  be  used 
during  test  in  the  same  way  they  would  be 
used  in  the  field.  For  example,  equipment 
designed  to  be  run  on  portable  generators 
should  be  powered  by  the  portable  generators 
throughout  the  test.  Secondary  failure 
problems  induced  by  generator  power  fluctua¬ 
tions  may  not  be  observed  if  the  system  is 
run  on  the  base  electrical  system. 

e.  The  maintenance  data  collection  (MDC) 
system  data  may  not  provide  a  true  reliabil¬ 
ity  picture.  Computer  halts,  for  example,  are 
not  reported  on  some  systems  when  down¬ 
time  is  less  than  10  minutes.  Test  team 
personnel  should  ensure  all  factors  that 
influence  reliability  are  investigated. 

f.  Reliability  reports  are  not  always  prop¬ 
erly  controlled.  R&M  data  products  by 
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themselves  may  not  be  classified,  but  when 
the  result  of  the  R&M  evaluation  depicts  a 
system's  operational  capability,  data  may  be 
classified.  AFOTEC  personnel  should  be 
familiar  with  the  system's  security  classifica¬ 
tion  guide. 

g.  There  is  often  misunderstanding  of  the 
method  used  to  compute  weapon  system  or 
mission  reliability.  It  is  imperative  AFOTEC, 
the  SPO,  and  the  user  clearly  understand  the 
method,  formulas,  parameters,  failure  defini¬ 
tion,  etc.,  early  in  the  test  planning  process. 

h.  OT&E  on  systems  with  high  reliabil¬ 
ity  present  challenges  to  the  test  planner. 
Several  methods  are  being  investigated; 
however,  a  single  policy  in  this  area  is 
lacking.  Test  planners  should  still  develop 
an  approach/concept  and  present  it  to  the 
TSG.  Among  the  methods  studied  are  use  of 


Bayesian  reliability  test  designs,  redefining 
reliability  requirements  so  testable  criteria 
can  be  used/developed,  and  focusing  the  test 
design  on  maintainability  versus  reliability. 

i.  Use  of  confidence  intervals/limits  on 
reliability  measurements  is  often  misunder¬ 
stood  by  those  unfamiliar  with  statistical 
estimation.  When  used,  they  should  be 
clearly  explained  so  the  decision  maker  will 
have  an  understanding  of  the  information 
being  provided  in  the  final  report.  For 
example,  it  may  be  better  to  state  how 
confident  we  are  the  user’s  requirement 
has/has  not  been  met  rather  than  giving  a 
confidence  interval  about  the  observed  'or 
projected)  reliability  measurement.  There  are 
no  current  policy  or  directives  on  applying 
confidence  levels  to  projected  reliability 
measurements. 
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11-1.  General: 

a.  Maintainability  and  reliability  are  the 
vo  major  system  characteristics  that  impact 

the  commonly  used  index — availability. 
While  maintainability  is  important  as  a 
factor  of  availability,  it  also  merits  substan¬ 
tial  consideration  as  an  individual  system 
characteristic.  Maintainability  is  a  factor  of 
the  design  process  and  an  inherent  design 
characteristic  that  is  quantitative  and  quali¬ 
tative  in  nature  and  therefore  lends  itself  to 
specification,  demonstration,  and  tradeoff 
analysis. 

b.  In  assessing  maintainability,  data 
elements  such  as  maintenance  time,  direct 
maintenance  man-hours,  and  system  down¬ 
time  are  collected.  The  data  are  then  re¬ 
ported  as  averages,  either  divided  by  some 
operational  base  such  as  flying  hours,  sorties, 
or  maintenance  events/actions,  or  categorized 
by  systems  to  highlight  the  areas  most 
needing  attention.  The  key  to  effective  main¬ 
tainability  assessment  is  first  to  measure 
maintainability  and  then  to  identify  those 
factors  which  are  influencing  the  results. 

c.  During  an  OT&E,  quantitative  and 
qualitative  aspects  of  maintainability  charac¬ 
teristics  are  addressed.  Quantitative  param¬ 
eters  for  maintainability  evaluation  can  be 
expressed  as  maintenance  downtime  per 
sortie,  as  a  usage  rate  of  manpower  resources 
(for  example,  maintenance  man-hours  per 
flying  hour),  as  the  total  required  manpower 
(maintenance  man-hours  per  operational 
unit),  as  a  time  to  restore  a  system  to  opera¬ 
tional  status  (mean  downtime),  etc.  Qualita¬ 
tive  aspects  of  maintainability  include  acces¬ 
sibility,  serviceability,  ease  or  difficulty  of 
maintenance,  safety,  and  human  factors 
associated  with  maintenance  actions.  These 
factors  affect  the  quantity,  skill  levels,  and 
specialty  codes  of  maintenance  personnel  and 
the  test  equipment  required  to  maintain  the 
system.  Qualitative  evaluations  of  maintain¬ 
ability  are  usually  done  by  experienced 
maintenance  technicians  using  subjective 
judgment  and  are  supported  by  quantitative 
maintainability  values. 

11-2.  Definitions  and  Concepts: 

a.  Maintainability.  The  measure  of  the 
ability  of  an  item  to  be  retained  in  or  re¬ 
stored  to  a  specified  condition  when  mainte¬ 
nance  is  performed  by  personnel  having 
specified  skill  levels,  using  prescribed  proce¬ 
dures  and  resources  at  each  prescribed  level 


of  maintenance  and  repair  (MIL-STD  7210. 
A  commonly  used  working  definition  states 
maintainability  is  the  inherent  characteristic 
of  a  design  that  determines  the  type  and 
amount  of  maintenance  required  to  retain 
that  design  in,  or  restore  it  to,  a  specified 
condition. 

b.  Maintenance.  All  actions  required  to 
retain  am  item  in,  or  restore  it  to,  a  speci¬ 
fied  condition.  This  includes  servicing, 
diagnosis,  repair,  modification,  modernization, 
overhaul,  rebuild,  test,  reclamation,  inspec¬ 
tion,  and  condition  determination. 

(1)  Preventive  Maintenance.  Systematic 
inspection,  detection,  and  correction  of  incip¬ 
ient  failures  either  before  they  occur  or 
before  they  develop  into  major  defects. 
Adjustment,  lubrication,  and  scheduled  checks 
are  included  in  the  definition  of  preventive 
maintenance. 

(2)  Corrective  Maintenance.  Maintenance 
performed  on  an  unscheduled  basis  to  restore 
equipment  to  satisfactory  condition  by  cor¬ 
recting  a  malfunction. 

(3)  Off-Equipment  Maintenance.  In-shop 
maintenance  performed  on  removed  compo¬ 
nents,  except  complete  aircraft  engines. 

(4)  On-Equipment  Maintenance.  Mainte¬ 
nance  accomplished  on  complete  end  items 
such  as  aircraft,  trainers,  support  equipment, 
CEM  equipment,  complete  round  munitions, 
and  complete  aircraft  engines. 

c.  Maintenance  Concept.  A  descrip¬ 
tion  of  the  essential  elements,  requirements 
considerations,  and  constraints  for  support  of 
a  new  weapon  system  or  equipment  in  its 
intended  operational  environment.  It  is 
prepared  LAW  AFR  66-14  to  be  an  integral 
part  of  the  ORD  required  by  AFR  57-1.  The 
maintenance  concept  forms  the  basis  for  all 
logistics  planning  and,  along  with  the  opera¬ 
tional  concept,  establishes  the  framework  for 
design. 

11-3.  Policy.  DODI  5000.2  and  AFR  800- 
18  outline  the  policy  for  implementing  and 
managing  maintainability  programs  for 
systems,  subsystems,  and  equipment.  Main¬ 
tainability  is  measured  during  the  entire 
T&E  effort.  During  DT&E,  test  conditions 
and  procedures  for  verification  demonstra¬ 
tions  attempt  to  reflect  the  operational  condi¬ 
tions  as  closely  as  possible.  During  OT&E, 
maintainability  assessments  include  evaluat¬ 
ing  test  results  against  criteria  expressed  in 
operational  terms  and  evaluating  logistics 
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supportability  of  the  system  against  mission 
requirements. 

11-4.  Considerations  in  Planning  Main¬ 
tainability  Assessment.  An  understanding 
of  the  principal  elements  of  maintainability 
is  essential  to  the  evaluation  planning  proc¬ 
ess.  The  factors  which  affect  the  ability  to 
perform  maintenance  are  the  design  of  the 
system,  the  technicians  performing  the  main¬ 
tenance,  and  the  logistics  support  concept. 
Another  factor  which  influences  the  quantita¬ 
tive  maintainability  measures  is  the  fre¬ 
quency  with  which  maintenance  is  required. 

a.  The  system  design,  both  hardware  and 
software,  affects  the  speed  and  ease  with 
which  maintenance  can  be  performed.  Ex¬ 
amples  of  these  effects  are  accessibility, 
visibility,  interchangeability,  and  simplicity. 

b.  Maintenance  personnel  can  have  a 
significant  impact  on  maintainability  assess¬ 
ments.  The  considerations  here  include  the 
experience  of  the  technicians,  training,  skill 
level,  supervision,  techniques  used,  physical 
coordination  and  strength,  number  of  techni¬ 
cians,  and  teamwork  requirements.  An  effort 
should  be  made  to  construct  a  test  team  that 
is  representative  of  an  operational  mainte¬ 
nance  unit  and  in  line  with  the  maintenance 
concept  of  the  system  under  test. 

c.  Some  logistics  support  considerations 
that  affect  maintainability  include  technical 
data  (TOs  and  manuals),  support  equipment, 
integrated  diagnostics,  and  sparing  concepts. 
These  factors  are  evaluated  separately  under 
"logistics  supportability"  (chapter  3)  and 
"integrated  diagnostics"  (chapter  13);  how¬ 
ever,  they  can  affect  repair  times,  downtimes, 
and  manpower  requirements.  Since  it  is 
rarely  possible  to  test  with  all  the  logistics 
support  elements  in  place,  adequate  evalua¬ 
tion  of  maintainability  may  require  simula¬ 
tion  of  the  operational  environment. 

d.  When  calculating  average  downtimes, 
repair  times  and  man-hours,  the  frequency 
with  which  different  maintenance  tasks  are 
required  can  have  a  significant  impact.  This 
maintenance  frequency  is  affected  by  equip¬ 
ment  reliability  and  the  preventive  mainte¬ 
nance  schedule.  There  should  be  sufficient 
test  exposure  to  allow  a  variety  of  mainte¬ 
nance  actions  to  be  required  during  test, 
thereby  achieving  more  accurate  repair 
frequency,  repair  times,  and  troubleshooting 
times.  These  are  some  of  the  reasons  main¬ 
tainability  demonstrations  can  only  supple¬ 
ment  and  not  replace  operational  testing. 
Demonstrations  can  be  used  to  quantify 
maintenance  times  for  tasks  that  will  not  be 
required  during  the  course  of  the  test.  These 
will  be  discussed  in  further  detail  in  section 


11-10. 

11-5.  Maintainability  Measures.  The 
following  paragraphs  describe  the  various 
MOEs  used  to  quantify  maintainability. 
Selection  of  MOEs  should  reflect  ORD  re¬ 
quirements. 

a.  Fix  Rate.  Percent  of  aircraft  which 
return  "code  3"  that  must  be  repaired  in  a 
specified  number  of  clock  hours  (AFP  57-9). 

b.  Maintenance  Personnel  Per  Opera¬ 
tional  Unit  (MP/OU).  The  number  of 
maintenance  personnel  that  will  be  required 
to  support  an  operational  unit  (excluding 
depot  level  and  other  manpower  that  is 
excluded  from  maintenance  planning  factors ) 
under  specified  operating  and  maintenance 
concepts.  The  user  of  this  term  needs  to 
define  the  operational  unit.  The  numbers  of 
maintenance  personnel  can  be  computed 
through  the  use  of  simulation  by  operating 
command  standards  or  maintenance  man¬ 
hour  per  flying  hour  calculations  (AFP  57-9). 

c.  Maintenance  Man-Hours  Per  Life 
Unit  (Operating  Hours,  Flight  Hours, 
Sorties)  (MMH/LU).  The  cumulative  man¬ 
hours  of  maintenance  expended  in  direct 
labor  during  a  given  period  of  time,  divided 
by  the  cumulative  number  of  end-item  life 
units  during  the  same  time.  The  MMH/LU 
is  expressed  at  each  level  of  maintenance 
and  summarized  for  all  levels  of  maintenance 
combined.  Corrective  and  preventive  main¬ 
tenance  is  included.  Man-hours  for  off-equip¬ 
ment  repair  of  replaced  components  and 
man-hours  for  daily  operational  checks  can 
be  included  for  some  systems.  The  life  unit 
must  be  clearly  defined  (AFP  57-9). 

_d.  Mean  Active  Maintenance  Time 
(M).  A  common  (not  necessarily  standard) 
term  defined  as  the  average  elapsed  time 
required  to  perform  scheduled  (preventive) 
and  unscheduled  (corrective)  maintenance. 
It  excludes  logistics  delay  time  and  adminis¬ 
trative  delay  time  and  is  expressed  as: 

M  -  QXMRT)  +  (fXMPT) 

*+  f 

Where  ^  is  the  corrective  maintenance  rate 
or  failure  rate,  f  is  the  preventive  mainte¬ 
nance  rate,  MRT  is  defined  in  paragraph 
ll-5(g),  and  MPT  is  the  mean  preventive 
maintenance  time. 

e.  Mean  Downtime  (MDT).  The  aver¬ 
age  elapsed  time  between  loss  of  mission 
capable  status  and  restoration  of  the  system 
to  mission  capable  status  (AFP  57-9).  Down¬ 
time  includes  maintenance  and  supply  (or 
logistics)  delay  time  (LDT),  administrative 
delay  time  (ADT),  and  actual  on-equipment 
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repair  (expressed  as  mean  active  mainte¬ 
nance  time,  M,  defined  above).  A  simple 
expression  for  MDT  is: 

MDT  =  M  +  LDT  +  ADT 

MDT  is  significantly  influenced  by  logistics 
support  and  maintenance  management  policy. 
Where  such  an  impact  is  noted,  its  effect  on 
MDT  must  be  separated  from  system  main¬ 
tainability  characteristics  and  reported.  MDT 
should  not  be  directly  measured  during 
IOT&E  because  support  elements  will  not  be 
representative  of  the  intended  operational 
environment.  Instead,  simulation  should  be 
used  to  calculate  MDT. 

f.  Mean  Man-Hours  to  Repair  (MMR). 
The  total  corrective  base  level  man-hours 
divided  by  the  total  corrective  maintenance 
events  for  a  given  period  of  time. 

g.  Mean  Repair  Time  (MRT).  The 
average  on-  or  off-equipment  corrective  main¬ 
tenance  time  in  an  operational  environment 
(AFP  57-9).  MRT  includes  all  maintenance 
actions  required  to  correct  the  malfunction, 
including  preparation  for  test,  troubleshoot¬ 
ing,  removal  and  replacement  of  components, 
repair,  adjustment,  functional  check,  etc. 
MRT  does  not  include  maintenance  or  supply 
delays  and  elapsed  time  for  preventive  main¬ 
tenance.  Hence  this  index  does  not  provide 
a  complete  measure  of  the  total  maintenance 
burden.  MRT  is  similar  to  mean  time  to 
repair  (MTTR)  or  mean  corrective  time 
(MCT),  but  is  referred  to  as  MRT  when  used 
as  an  operational  term  to  avoid  confusion 
with  the  frequently  used  contractual  term  of 
MTTR. 

h.  Mission  Time  to  Restore  Functions 
(MTTRF).  A  measure  of  mission  maintain¬ 
ability.  MTTRF  is  the  total  corrective  critical 
failure  maintenance  time,  divided  by  the 
total  number  of  critical  failures,  during  the 
course  of  a  specified  mission  profile  (MID- 
STD  721). 

i.  Mean  Time  to  Restore  System 
(MTTRS).  A  measure  of  system  maintaina¬ 
bility  related  to  availability  and  readiness. 
The  total  corrective  maintenance  time,  associ¬ 
ated  with  downing  events,  during  a  stated 
period  of  time.  (Excludes  time  for  off-equip¬ 
ment  maintenance  and  repair  of  detached 
components.) 

j.  Mean  Time  to  Service  (MTTS).  A 
measure  of  an  on-system  maintainability 
characteristic  related  to  servicing  that  is 
calculated  by  dividing  the  total  scheduled 
crew/operator  servicing  time  by  the  number 
of  times  the  item  was  serviced. 

k.  Turnaround  Time  (TAT).  The  main¬ 
tenance  time  needed  to  prepare  an  aircraft 


system  for  another  sortie/mission.  The  time 
begins  at  engine  shutdown  or  landing  and 
includes  servicing,  preflight/postflight  inspec¬ 
tion  delays,  unscheduled  maintenance,  and 
reconfiguration  activities. 

1.  Quick  Turnaround  Time  (QTAT). 
QTAT  is  an  indicator  of  the  maximum  surge 
sortie  generation  capability  and  is  the  short¬ 
est  total  elapsed  time  required  to  prepare  an 
aircraft  returning  from  one  mission  for  anoth¬ 
er  mission  in  the  same  configuration.  Per¬ 
sonnel,  equipment,  and  expendables  (such  as 
munitions)  will  be  pre-positioned.  The  muni¬ 
tions  load  may  be  changed  provided  recon¬ 
figuration  is  not  required.  Operational  and 
maintenance  policy  can  affect  QTAT.  Since 
quick  turnaround  procedures  require  typical 
maintenance  procedures,  special  tests  must 
be  arranged.  The  assessment  must  consider 
comparison  of  the  test  environment  to  the 
intended  operational  environment.  For 
multimission  aircraft,  more  than  one  QTAT 
test  may  be  required.  QTAT  may  be  asso¬ 
ciated  with  the  term  "integrated  combat  turn¬ 
around  (ICT)."  ICT  is  an  authorized  excep¬ 
tional  servicing  operation  for  tactical  aircraft 
during  which  the  simultaneous  fueling, 
munitions  loading/tanloading  general  servic¬ 
ing,  and  other  specific  maintenance  actions 
are  performed  (TACR  66-5). 

mu  Mean  Time  to  Perform  Munitions/ 
Missile  Generation  Functions.  Mean  time 
to  repair,  assemble,  deliver,  or  load  are  direct 
time  measurements  used  to  determine  the 
ability  of  the  system  to  support  user  genera¬ 
tion  time  requirements.  The  mean  time  to 
perform  the  various  functions  is  calculated  by 
dividing  the  total  time  required  to  perform 
each  function  (assembly,  delivery,  or  loading) 
by  the  total  number  of  times  each  functions 
was  performed  during  test.  Criteria  for  these 
functions  are  normally  contained  in  the 
system  maintenance  concept.  These  measure¬ 
ments  are  used  as  data  for  input  to  the 
system  availability  model  if  one  is  used. 

n.  Off-System  Maintainability  Indices. 
Off-equipment  measures  are  particularly  im¬ 
portant  if  a  system’s  maintenance  concept 
involves  extensive  use  of  modular  removal 
and  replacement,  since  this  type  of  concept 
transfers  the  maintenance  burden  to  off- 
equipment  maintenance.  Off-equipment 
maintainability  measures  are  essential  to 
assess  combat  environment  off-equipment 
repair  and  logistics  capability  to  maintain  the 
system.  Off-equipment  parameters  could 
include  time  to  repair  at  intermediate  and 
depot  levels  or  repair  man-hours  for  off- 
equipment  repair  and  indirect  man-hours 
required  to  support  the  system.  Other  in¬ 
dices  may  be  used  as  required  to  address  the 
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total  maintenance  burden  of  the  system. 

o.  Qualitative  Maintainability  Char¬ 
acteristics.  Qualitative  aspects  of  main¬ 
tainability  include  accessibility,  serviceability, 
ease  of  maintenance,  safety,  and  human 
factor  requirements.  The  intent  is  to  identify 
the  qualitative  aspects  of  maintainability  that 
significantly  influence  the  quantitative  meas¬ 
ures  of  maintainability.  Assessment  of  these 
aspects  affords  an  opportunity  to  address  a 
broad  range  of  maintenance  concerns  and 
their  interactive  effect  on  maintenance  re¬ 
sources. 

11-6.  Integrated  Diagnostics.  One  aspect 
of  maintainability  that  has  received  signifi¬ 
cant  attention  in  recent  system  designs  is 
integrated  diagnostics.  This  includes  both 
internal  or  automated  diagnostic  systems, 
referred  to  as  a  built-in  test  (BIT),  and 
external  diagnostic  systems,  referred  to  as 
technical  documentation  automatic  test  equip¬ 
ment  (ATE),  test  sets,  or  off-line  test  equip¬ 
ment.  Chapter  13  contains  a  detailed  discus¬ 
sion  of  this  area. 

11-7.  Maintainability  Growth: 

a.  DODI  5000.2  states  that  maintainability 
growth  is  required  during  full-scale  develop¬ 
ment,  concurrent  development  and  produc¬ 
tion,  and  during  initial  deployment.  Pre¬ 
dicted  maintainability  is  stated  as  series  of 
intermediate  milestones,  with  associated  goals 
and  thresholds,  for  each  contractually  speci¬ 
fied  parameter  for  each  of  those  phases.  A 
period  of  testing  is  scheduled  in  conjunction 
with  each  intermediate  milestone.  A  block 
of  time  and  resources  are  scheduled  for  the 
correction  of  deficiencies  and  defects  found 
during  the  testing  to  prevent  their  recurrence 
in  the  operational  inventory.  Approved 
maintainability  growth  requirements  are 
assessed  and  briefed  at  decision  milestones. 

b.  As  with  reliability,  a  successful  main¬ 
tainability  growth  program  depends  on  a  test 
program  that  subjects  the  system  to  test 
exposure  adequate  enough  to  uncover  main¬ 
tainability  design  deficiencies.  This  is  usual¬ 
ly  accomplished  through  maintainability 
verification  and  demonstration  activities.  To 
achieve  growth,  the  inadequate  design  fea¬ 
tures  must  be  analyzed,  modifications  incor¬ 
porated,  and  the  modified  system  retested  to 
verify  the  validity  of  the  fix.  It  is  this  test- 
analyze-fix-test  (TAFT)  philosophy  that  is  the 
basis  for  maintainability  growth.  However, 
there  are  other  ways  to  improve  maintain¬ 
ability.  Good  training  (including  hands-on 
experience)  is  one  way  to  learn.  As  tasks 
are  repeated  a  learning  process  occurs.  This 
process  can  be  easily  modeled. 


11-8.  Tracking  Maintainability  Growth: 

a.  The  objectives  of  growth  tracking  are 
to  determine  if  growth  is  occurring  and  to 
what  degree,  to  estimate  the  maintainability 
parameter  values,  and  to  formulate  a  projec¬ 
tion  of  these  values. 

b.  The  program  manager  must  establish 
contractual  and  mature  operational  maintain¬ 
ability  threshold  and  goal  values  and  main¬ 
tain  traceability  throughout  the  acquisition 
process.  These  values  are  updated  during 
the  acquisition  process  and  must  be  pre¬ 
sented  by  the  program  manager  at  appropri¬ 
ate  milestones.  Usually  the  predicted  main¬ 
tainability  growth  is  shown  as  series  of 
intermediate  points  to  be  attained  during  the 
system’  acquisition. 

c.  During  OT&E,  the  concept  of  quantify¬ 
ing  maintainability  growth  is  useful  in  deter¬ 
mining  whether  or  not  mature  operational 
requirements  for  maintainability  can  be 
achieved.  System  design  changes  for  main¬ 
tainability  are  intended  to  improve  the  time 
required  to  repair  or  restore  an  item  to  a 
specified  condition.  However,  such  changes 
may  in  fact  degrade  maintainability.  Unlike 
reliability,  there  is  no  standard  technique 
used  to  track  maintainability  growth  based 
on  developmental/operational  test  data  with 
consideration  given  to  planned  design,  proce¬ 
dures,  and/or  training  improvements.  At 
publication  of  this  pamphlet,  an  effort  to 
study  maintainability  prediction  methods  and 
adopt  a  set  for  use  by  AFOTEC  was  under 
consideration.  Results  of  the  study  will  be 
published  in  future  updates  to  this  pamphlet. 
In  the  interim,  the  following  approaches  have 
been  used: 

(1)  Compare  tested  system  data  with 
data  from  a  similar  fielded  system.  Use 
expert  judgment  to  predict  future  maintain¬ 
ability  of  tested  system. 

(2)  Use/modify  methods  described  in  MIL- 
HDBK  472,  Maintainability  Prediction. 

(3)  Vary  maintainability  related  input  to 
operational  availability  models/simulations 
to  determine  impacts  (if  any)  of  achieving  (or 
not  achieving)  mature  maintainability  values. 

(4)  Formulate  a  progress  function  or 
learning  curve  based  on  the  fact  the  time 
required  to  maintain  an  item  drops  as  the 
total  number  of  items  maintained  doubles. 

11-9.  Data  Analysis: 

a.  Most  data  elements  for  maintainability 
MOEs  are  the  same  for  IOT&E  and  FOT&E. 
Administrative  and  logistics  delay  factors 
may  vary.  During  IOT&E,  these  delay 
factors  must  be  estimated  (e.g.,  from  data  on 
similar  weapon  systems);  during  FOT&E, 
they  can  usually  be  measured  directly. 
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Common  data  elements  ai  : 

(.1)  Turnaround  times. 

(2)  Number  and  type  of  different  missions 
or  system  configurations. 

(3)  Assembly,  repair,  and/or  loading 
times. 

(4)  Not-mission-capable  hours. 

(5)  Logistics  delay  times. 

(6)  Administrative  delay  times. 

(7)  Number  of  repair  actions. 

(8)  Number  of  flying  hours. 

19)  Number  of  sorties. 

(10)  Number  of  personnel  performing 
tasks. 

(11)  Questionnaires. 

b.  Data  sources  for  maintainability  are 
similar  to  those  for  availability  and  reliabil¬ 
ity- 

c.  The  test  team  will  generally  collect 
repair  data  on  failures  during  the  test  period 
in  order  to  compute  the  quantitative  param¬ 
eters  such  as  model  output,  MTTRS,  and 
MRT.  MTTRS  or  MRT  is  usually  calculated 
for  the  system  and  each  subsystem/LRU. 
The  system  MDT,  while  useful  for  determin¬ 
ing  system  availability,  does  not  adequately 
identify  problem  areas.  Therefore,  subsys¬ 
tem/LRU  level  MRT  should  be  analyzed  to 
highlight  repair  times  that  are  inordinately 
high  due  to  design  or  other  characteristics, 
and  to  indicate  undesirable  trends.  Adminis¬ 
trative  and  logistics  delay  times  and  preven¬ 
tive  maintenance  times  should  also  be  ex¬ 
amined  for  unusually  high  times  or  adverse 
trends.  The  distribution  of  the  individual 
repair  times  can  be  determined  by  using 
histograms  and  goodness-of-fit  tests.  If 
limited  data  are  available  for  statistical 
confidence,  a  lognormal  distribution  is  usual¬ 
ly  assumed. 

d.  Qualitative  maintainability  data  col¬ 
lected  by  the  test  team  will  be  analyzed  to 
identify  problems  with  equipment  design, 
installation,  accessibility,  or  servicing.  Spe¬ 
cial  emphasis  should  be  placed  on  equipment 
design  problems  that  could  lead  to  main¬ 
tenance  errors  or  safety  hazards.  Repair 
times  that  are  excessive  should  be  further 
analyzed  to  determine  the  primary  cause. 
The  results  of  these  analyses  are  usually 
combined  with  the  logistics  supportability 
data  for  evaluation  purposes. 

11-10.  Maintainability  Demonstrations 
(M-demos).  Maintainability  demonstrations, 
or  M-demos,  are  normally  associated  with 
DT&E.  These  are  conducted  by  the  system 
contractor  to  demonstrate  compliance  with 
specifications.  M-demos  can  be  used  to 
acquire  data  during  OT&E  if  done  in  the 
operational  environment.  M-demos  performed 


during  OT&E,  or  operational  M-demos.  are 
"staged”  maintenance  events  done  in  an 
operational  environment  to  obtain  quantita¬ 
tive  maintainability  information  not  otherwise 
available  during  OT&E.  These  are  some¬ 
times  described  as  ease-of-maintenance  dem¬ 
onstrations  (removal  and  replacement  of 
components  or  performance  of  tasks;.  Other 
potential  M-demos  can  be  performed  by 
intentionally  inserting  faulty  components  into 
the  system  to  assess  troubleshooting  and 
repair  capability.  This  is  especially  impor¬ 
tant  when  testing  highly  reliable  systems 
(test  exposure  small  compared  with  expected 
mean  time  between  failure).  The  logistics 
evaluation  manager  should  decide  on  the 
method  of  acquiring  the  data  through  estab¬ 
lishment  of  a  formal  requirement  with  the 
SPO  to  ensure  a  requirement  is  included  in 
contract  for  acquiring  data  through  the  use 
of  M-demos.  The  logistics  manager  should 
become  familiar  with  MIL-STDs  470  and  471 
and  identify  the  test  data  requirements. 
Many  of  these  M-demos  can  only  be  assess¬ 
ments,  not  evaluations.  This  is  because  they 
do  not  necessarily  relate  exactly  to  the  fre¬ 
quency  and  mode  of  failures,  and  they  do  not 
give  data  on  ’’induced"  and  "no  defect"  fail¬ 
ures.  The  test  team  should  use  the  approved 
integrated  logistics  support  plan  to  identify 
the  logistics  support  required.  When  per¬ 
forming  M-demos,  certain  considerations 
apply.  M-demos  should: 

a.  Be  clearly  defined  and  scoped  in  the 
OT&E  test  plan. 

b.  Be  coordinated  with  the  implementing 
command. 

c.  Be  performed  at  or  near  the  end  of 
operational  test  so  as  not  to  interfere  with  or 
alter  the  equipment  under  test. 

d.  Be  conducted  in  an  environment  which 
simulates,  as  closely  as  possible,  the  opera¬ 
tional  and  maintenance  environment  planned 
for  the  item  (i.e.,  blue-suit  maintenance  with 
no  contractor  involvement).  The  environment 
should  be  representative  of  the  working 
conditions,  tools,  support  equipment,  spares, 
facilities,  and  technical  publications  that 
would  be  required  during  operational  service 
as  described  in  the  maintenance  plan. 

e.  Be  videotaped. 

f.  In  conjunction  with  the  ease-of-mainte- 
nance  demonstration,  the  test  team  should 
use  the  approved  integrated  logistics  support 
plan,  when  required  and  established  by  the 
contractor,  scaled  to  the  number  of  test  items 
employed  in  the  demonstration,  to  identify 
the  logistics  support  required. 

g.  Ease-of-maintenance  demonstrations 
should  be  videotaped. 
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11-11.  Lessons  Learned: 

a.  There  may  not  be  a  requirement  for 
the  contractor  to  provide  maintainability  data 
to  Air  Force  agencies,  thereby  limiting  the 
amount  of  data  available  to  AFOTEC  person¬ 
nel  during  test.  Review  contract  deliverables 
before  test  planning  activities,  preferably 
before  contract  award,  to  determine  AFOTEC 
accessibility  to  contractor  data.  See  the 
AFOTEC  LGOI  on  requests  for  proposal 
(RFP)  as  an  aid  in  this  review. 

b.  During  early  phases  of  test  programs, 
the  system  may  be  maintained  by  contractor 
personnel  or  by  a  mix  of  contractor  and  blue- 
suit  personnel.  The  contractor  personnel  may 
be  documenting  maintenance  actions  using 
their  internal  data  collection  system  while 
the  blue-suitors  are  using  SEDS  or  the  MDC 
system  to  document  their  maintenance  ac¬ 
tions.  There  is  often  no  direct  link  between 
the  system,  which  results  in  data  loss. 
AFOTEC  should  make  every  attempt  to 
establish  a  common  data  collection  system. 

c.  A  lack  of  common  definition  of  main¬ 
tainability  terms  among  DT&E,  OT&E,  and 
contractor  personnel  has  caused  problems  in 
the  past.  AFOTEC  should  insist  on  an  early 
establishment  of  JRMET  to  set  the  ground 


rules  early  in  the  test  program. 

d.  Test  results  can  be  distorted  if  equip¬ 
ment  is  not  maintained  by  maintenance 
personnel  representative  of  using  command 
personnel.  Additionally,  problems  in  han¬ 
dling  large  or  heavy  objects  will  be  more 
apparent  if  personnel  who  are  average  in  size 
and  strength  are  used  to  support  the  system 
during  test.  Particular  attention  must  be 
given  to  these  areas  to  ensure  accurate 
analysis  and  reporting. 

e.  Maintenance  data  collection  may  be 
inadequate  during  the  early  phase  of  test 
programs  because  a  maintenance  analyst  is 
not  available.  This  results  in  incomplete  and 
less  than  meaningful  data.  AFOTEC  should 
ensure  that  a  maintenance  analyst  is  as¬ 
signed  to  the  test  team  before  the  test  starts. 

f.  Maintainability  demonstrations  may  be 
conducted  under  conditions  that  are  not 
representative  of  field  conditions  resulting  in 
questionable  data  for  OT&E  purposes. 
AFOTEC  personnel  should  try  to  influence 
the  SPO  to  conduct  the  demonstrations  under 
representative  field  conditions.  At  the  very 
least,  AFOTEC  personnel  should  audit  the 
data  to  determine  their  usefulness. 
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Chapter  12 

SOFTWARE  EVALUATION 


12-1.  Introduction.  Software  is  an  inte¬ 
gral  part  of  almost  every  weapon  system. 
Because  it  crosses  many  disciplines,  poorly 
designed  software  can  render  a  weapon 
system  ineffective  and  make  the  system 
difficult  to  support.  The  test  methodologies 
in  the  following  paragraphs  were  developed 
to  determine  the  contribution  of  software  to 
a  system’s  operational  effectiveness  and 
suitability. 

a.  Software  Effectiveness.  Software 
effectiveness  or  performance  evaluation  (from 
an  OT&E  standpoint)  is  always  done  within 
the  context  of  overall  system  performance 
evaluation  in  an  operational  environment. 
The  evaluation  of  system  functions,  whether 
implemented  through  hardware  or  software, 
falls  into  the  area  of  operational  effective¬ 
ness.  There  are  no  upper-level,  agreed-upon 
metrics  that  characterize  the  nature  of  soft¬ 
ware  performance  in  its  operational  con¬ 
figuration;  however,  two  software  suitability 
indicators,  software  maturity,  and  software 
usability  do  provide  insight  into  software’s 
impact  on  mission  performance.  For  in¬ 
stance,  portions  of  the  software  maturity 
trend  analysis  provide  system  and  subsystem 
level  trend  information  about  the  progress  of 
software  performance,  while  a  software 
usability  evaluation  provides  information 
regarding  the  user-machine  interface  and  the 
user-friendliness  of  the  software.  Thus  in 
OT&E,  software  effectiveness  focuses  on 
software  problems  and  the  effects  of  those 
problems  on  system  operation.  The  head¬ 
quarters  and  test  team  software  evaluation 
personnel  provide  consultation  to  other  test 
team  members  in  evaluating  the  contribution 
of  software  to  system  effectiveness  and  assist 
in  defining  test  scenarios  for  the  system  that 
exercise  known  or  suspected  weak  areas  in 
system  design. 

b.  Software  Suitability.  The  Software 

Analysis  Division’s  primary  area  of  focus  is 
on  software  suitability.  Software  suitability 
encompasses  a  number  of  software  support 
activities  and  functions.  Software  maturty, 
software  usability,  and  software  support- 
ability  are  important  indicators  of  software 
suitability.  Software  supportability  is  a 
subset  of  software  suitability  and  has  four 
primary  areas  for  evaluation:  software 

maintainability,  software  support  life  cycle 
processes,  software  support  resources,  and 
software  support  risk  assessment  Related 
areas  of  evaluation  include  spare  computing 


capacity,  computer  security,  software  reliabil¬ 
ity,  and  adaptive  software  suitability  assess¬ 
ments.  The  details  of  each  type  of  evalua¬ 
tion  are  provided  below.  A  handbook  has 
been  published  by  AFOTEC/LG5  which 
provides  detailed  guidance  on  how  to  report 
the  results  of  these  evaluations. 

12-2.  Software  Maturity  Evaluation. 
Software  maturity  is  a  measure  of  the  soft¬ 
ware’s  progress  in  its  evolution  toward  satis¬ 
fying  all  documented  user  requirements. 
Software  maturity  trend  analysis  can  be 
useful  in  three  areas:  readiness  for  IOT&E, 
software  suitability,  and  software  effective¬ 
ness.  It  is  a  qualitative  evaluation  using 
software  change  trend  data  and  assessing 
both  the  timeliness  of  the  software  changes 
and  the  ability  of  the  software  to  evolve 
toward  "error- free,’’  effective  software. 

a.  Method.  This  evaluation  is  conducted 
using  the  guidelines  prescribed  in  AFOTECP 
800-2,  volume  6,  Software  Maturity  Assess¬ 
ment  Guide.  As  soon  as  software  comes 
under  formal  configuration  control,  software 
change  data  should  be  collected  by  the  pro¬ 
gram  office.  AFR  80-14  require  these  data 
be  shared  by  all  test  organizations.  All 
software  changes  (both  failures  or  enhance¬ 
ments)  are  plotted  cumulatively  over  time. 
Each  change  is  weighted  by  a  multiplier 
based  on  its  severity  (defined  in  DOD-STD 
2167A,  Defense  System  Software  Devel¬ 
opment,  appendix  C).  Also  cumulatively 
plotted  over  time  are  the  software  changes 
implemented.  Two  other  trend  measures  are 
also  collected:  the  trend  of  the  average  time 
required  to  make  changes  and  the  trend  of 
the  average  severity  of  changes  being  iden¬ 
tified. 

b.  Data  Requirements.  The  data  re¬ 
quirements  to  perform  this  evaluation  in¬ 
clude  software  changes  required,  software 
changes  implemented,  problem  severity,  and 
completeness  of  testing.  It  is  important  the 
need  for  software  maturity  data  be  discussed 
at  the  TPWG  and  arrangements  made  to 
obtain  it  from  the  development  contractor 
through  the  SPO.  Reference  attachment  8 
for  a  copy  of  our  generic  data  item  descrip¬ 
tion. 

c.  Evaluation.  The  test  team  deputy  for 
software  evaluation  fDSE)  and  the  head¬ 
quarters  software  evaluation  manager  ex¬ 
amine  the  curves  produced  from  the  software 
change  data.  In  a  mature  system,  the  curve 
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of  changes  required  should  flatten  to  some 
steady  state  value,  while  the  curve  of  chang¬ 
es  implemented  should  converge  with  the 
curve  of  changes  required  (see  figure  12-1). 
This  would  show  that  fewer  and  less  severe 
problems  are  being  found  and  that  software 
problems  are  being  corrected  faster  than  they 
are  being  discovered.  Trend  data  on  the 
time  required  to  make  software  changes  will 
indicate  how  well  the  development  contractor 
is  able  to  make  critical  software  changes 
while  the  average  severity  trend  will  indicate 
whether  or  not  major  software  problems  are 
still  being  identified  or  if,  indeed,  the  opera¬ 
tional  software  is  truly  maturing.  When 
examining  the  curves,  test  completeness  must 
also  be  considered  or  the  maturity  data  may 
be  misinterpreted.  Figure  12-2  is  a  com¬ 
posite  presentation  of  the  software  maturity 
trend  information  used  in  test  readiness 
determinations,  test  report  briefings,  and  in 
AFOTEC  final  reports.  In  determining 
software’s  readiness  to  support  IOT&E, 
maturity  trends  are  used  to  identify  the 
presence  of  a  stabilized  software  baseline 
and  an  absence  of  mission  critical  software 
problems.  Software  suitability  information  is 
in  the  form  of  how  accurately  and  efficiently 
software  problems  and  changes  are  developed 
and  implemented,  while  software  effective¬ 
ness  information  centers  on  the  identification 
of  software  changes  and  the  corresponding 
impact  on  system  functional  capabilities. 

d.  Assessment  Criteria.  Qualitative. 
This  evaluation  is  based  on  trend  informa¬ 
tion,  and  there  are  no  quantitative  evalua¬ 
tion  criteria. 

12-3.  Software  Usability  Evaluation. 
Software  usability  is  a  measure  of  the  man- 
machine  interface  between  the  operator  and 
the  software.  Software  usability  evaluates 
how  easy  the  software  is  to  use  by  a  typical 
system  operator.  The  evaluation  examines 
the  usability  attributes  of  confirmability, 
controllability,  workload  suitability,  descrip¬ 
tiveness,  consistency,  and  simplicity.  This 
evaluation  is  usually  performed  as  part  of  a 
larger  human  factors  evaluation  but,  under 
certain  circumstances,  may  be  a  stand-alone 
OT&E  objective. 

a.  Method.  This  evaluation  uses  the 
questionnaire  in  AFOTECP  800-2,  volume  4, 
Software  Usability  Evaluator’s  Guide.  The 
questionnaire  is  designed  to  be  administered 
to  operators  who  are  experienced  in  using 
the  operator-machine  interfaces.  Comments 
play  an  important  role  in  the  evaluation  and 
the  interviewer  should  also  solicit  comments 
from  the  operators. 

b.  Data  Requirements.  Requirements 


to  perform  this  evaluation  include  the  ques¬ 
tionnaire  in  AFOTECP  800-2,  vnlnm*  4; 
answer  sheets  completed  by  the  operators; 
and  the  operators'  comments. 

c.  Evaluation.  The  results  (numerical 
scores  and  histogram)  of  the  questionnaire 
are  analyzed  along  with  the  operator  com¬ 
ments  to  qualitatively  assess  the  usability 
strengths  and  weaknesses  of  the  interface. 

d.  Assessment  Criteria.  Qualitative. 
There  are  no  quantitative  evaluation  criteria 
for  this  evaluation. 

12-4.  Software  Supportability.  As  men¬ 
tioned  earlier,  there  are  four  primary  areas 
of  focus  that  contribute  to  the  overall  assess¬ 
ment  of  software  supportability;  software 
maintainability,  software  support  life  cycle 
processes,  software  support  resources,  and 
software  support  risk  assessment. 

a.  Software  Maintainability  Evalu¬ 
ation.  The  software  maintainability  evalua¬ 
tion  focuses  on  the  quality  of  the  computer 
program  source  code,  its  associated  documen¬ 
tation,  and  the  overall  design  implementa¬ 
tion  with  regard  to  facilitating  the  task  of 
later  changing  the  computer  software.  Soft¬ 
ware  changes  could  be  for  the  purpose  of 
correcting  errors,  adding  system  capabilities, 
deleting  functions,  or  modifying  software  to 
be  compatible  with  hardware  changes.  The 
software  maintainability  evaluation  measures 
the  extent  to  which  the  software  design,  as 
reflected  in  the  source  code  listings  and 
documentation,  has  good  maintainability 
characteristics.  These  characteristics  include 
modularity,  descriptiveness,  consistency, 
simplicity,  expendability,  testability,  trace- 
ability,  convention,  design,  and  organization. 

(1)  Method.  A  team  of  five  software 
evaluators  completes  structured  question¬ 
naires  contained  in  AFOTECP  800-2,  volume 
3,  Software  Maintainability  Evaluation 
Guide,  to  evaluate  software  documentation, 
design  implementation,  and  a  representative 
sample  of  software  source  code  (modules). 
These  evaluators,  usually  from  the  support¬ 
ing  or  using  command,  should  be  experienced 
software  mafn tamers,  typical  of  those  which 
will  have  responsibility  for  software  support 
of  the  system  when  the  Air  Force  assumes 
software  maintenance  responsibility. 

(2)  Data  Requirements.  Data  require¬ 
ments  to  perform  this  evaluation  are  all 
deliverable  software  documentation;  software 
source  code  listings;  questionnaires  contained 
in  AFOTECP  800-2,  volume  3;  answer  sheets 
completed  by  the  evaluators;  and  the  evalua¬ 
tors’  comments. 

(3)  Evaluation.  Evaluator  answer  sheets 
and  comments  are  collected,  scored,  and 
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Figure  1 2-2.  Software  Maturity  (Final  Report) 
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compared  to  determine  evaluation  results. 

To  determine  the  likelihood  the  software  will 
be  easily  maintained  or  not,  the  average 
score  from  the  questions  is  compared  to  a 
numerical  threshold  value  of  3.5  (on  a  scale 
of  6  good  to  1  bad).  Scores  less  than  3.5 
indicate  problem  areas  needing  further  anal¬ 
ysis.  Histogram  analysis  of  scores  may  also 
be  used  to  understand  and  report  the  signifi¬ 
cance  of  the  results.  The  DSE  or  headquar¬ 
ters  software  evaluation  manager  will  then 
further  examine  detailed  questionnaire  re¬ 
sults  and  comments  to  discover  the  strengths 
and  weaknesses  of  the  software’s  maintain¬ 
ability  characteristics. 

(4)  Assessment  Criteria.  There  are  no 
quantitative  evaluation  criteria  for  this 
evaluation;  although,  on  occasion,  some  using 
commands  have  stated  the  3.5  numerical 
threshold  as  their  operational  requirement. 

b.  Software  Support  Resources  Eval¬ 
uation.  Software  support  resources  include 
the  personnel,  computer  support  systems, 
configuration  management  system,  contin¬ 
gency  plans,  and  software  support  facilities 
required  by  the  depot  support  agency  to 
accomplish  software  modifications.  The  goal 
of  this  evaluation  is  to  assess  the  adequacy 
of  the  in-place  or  planned  software  support 
resources  to  satisfy  user  requirements  for 
postdeployment  software  support. 

(1)  Method.  The  questionnaire  in 
AFOTECP  800-2,  volume  5,  Software  Sup¬ 
port  Resources  Evaluation  Guide,  is  tailored 
for  the  particular  system  being  evaluated. 
The  questionnaire  is  administered  by  the 
DSE  or  software  evaluation  manager  in  one 
of  two  ways:  using  structured  interviews  or 
using  designated  software  evaluators.  If  the 
structured  interview  method  is  used,  the  DSE 
or  software  evaluation  manager  interviews 
managerial  and  technical  people  and  reviews 
software  support  documentation  to  gather 
information  to  subjectively  answer  the  ques¬ 
tionnaire.  This  method  is  most  effective 
during  the  system  planning  and  design 
stages.  The  software  evaluator  method  uses 
a  panel  of  software  support  resources  experts 
to  answer  the  questionnaire.  This  method  is 
best  suited  for  the  actual  operational  stage 
of  the  software  support  resources. 

(2)  Data  Requirements.  Requirements  to 
perform  this  evaluation  include  software 
support  documentation,  completed  tailored 
questionnaires,  and  evaluator  comments. 

(3)  Evaluation.  The  DSE  or  software 
evaluation  manager  reviews  the  completed 
questionnaire  with  the  comments  to  deter¬ 
mine  the  state  of  the  software  support  re¬ 
sources.  The  results  are  used  in  a  checklist 
fashion  to  identify  deficiencies.  The  com¬ 


ments  are  most  important  to  this  type  of 
evaluation  for  providing  insight  into  problem 
areas. 

(4)  Assessment  Criteria.  Qualitative. 
There  are  no  quantitative  evaluation  criteria 
for  this  evaluation. 

c.  Software  Support  Life  Cycle  Proc¬ 
ess  Evaluation.  The  software  support  life 
cycle  process  is  the  environment  in  which 
software  and  its  support  resources  are  pro¬ 
cured,  developed,  operated,  and  supported. 
This  evaluation  was  developed  to  determine 
the  degree  to  which  certain  software  develop¬ 
ment  and  support  management  procedures 
affect  a  particular  system.  The  major  ele¬ 
ments  of  the  software  support  life  cycle 
process  are  software  project  and  configura¬ 
tion  management.  Software  project  manage¬ 
ment  includes  planning,  organizational  struc¬ 
ture,  design  and  implementation  methods, 
test  strategies,  and  project  interfaces.  Soft¬ 
ware  configuration  management  includes  the 
control  of  software  changes  through  technical 
and  administrative  actions.  The  evaluation 
of  the  software  support  life  cycle  process  is 
applicable  to  all  phases  of  a  program.  For 
early  operational  assessments,  the  software 
development  process  is  a  major  focus  of  the 
software  evaluations.  This  evaluation  is 
structured  to  determine  if  significant  prob¬ 
lems  are  occurring  which  could  impact  the 
readiness  of  the  system  to  meet  its  OT&E 
schedule.  For  systems  in  OT&E,  this  evalua¬ 
tion  is  geared  toward  the  management  proc¬ 
ess  being  implemented  by  the  depot  software 
support  agency. 

(1)  Method.  The  DSE  or  software  evalu¬ 
ation  manager  uses  the  questionnaire  con¬ 
tained  in  AFOTECP  800-2,  volume  2,  Soft¬ 
ware  Support  Life  Cycle  Process  Evaluation 
Guide,  to  accomplish  this  evaluation.  This 
questionnaire  was  not  meant  to  be  completed 
in  one  sitting,  but  is  to  be  completed  over  a 
period  of  time  throughout  the  development 
and  operational  test  and  evaluation  cycle. 
The  questionnaire  is  used  as  a  guide  or 
checklist  when  reviewing  program  documen¬ 
tation  (e.g.,  PMP,  TEMP,  CRLCMP,  etc.)  or 
attending  meetings  (TPWG,  CRWG,  PDR, 
CDR,  etc.).  Specific  questions  are  then 
answered  based  on  information  gathered  from 
the  documentation  or  the  meetings.  Com¬ 
ments  are  written  after  each  question  just¬ 
ifying  the  evaluation.  The  evaluation  is 
updated  over  time  as  situations  change.  For 
systems  in  OT&E,  the  questionnaire  helps 
guide  an  assessment  of  the  computer  re¬ 
sources  life  cycle  management  plan 
(CRLCMP)  and  other  evolving  documents. 
The  final  element  of  the  evaluation,  however, 
is  an  assessment  of  the  eventual  software 
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support  agency  management  process  needed 
to  design,  implement,  and  control  post¬ 
deployment  software  changes. 

(2)  Data  Requirements.  Requirements 
to  perform  this  evaluation  include  program 
documentation,  information  from  meetings, 
life  cycle  management  plans,  and  the  com¬ 
pleted  questionnaire  with  supporting  com¬ 
ments. 

(3)  Evaluation.  The  DSE  or  software 
evaluation  manager  uses  the  questionnaire 
to  find  potential  problem  areas  in  program 
or  configuration  management. 

(4)  Assessment  Criteria.  Qualitative. 
There  are  no  quantitative  evaluation  criteria 
for  this  evaluation. 

d.  Bisk  Assessment  Methodology  for 
Software  Supportability  (RAMSS). 
RAMSS  is  a  risk  assessment  method  that 
provides  software  supportability  information 
in  terms  of  risks  for  those  systems  which 
are  dependent  on  computer  operations.  Risk 
is  defined  as  the  probability  of  not  accom¬ 
plishing  projected  user  software  change 
requirements  with  the  currently  scheduled 
resources  such  as  personnel,  equipment,  and 
facilities.  The  methodology  employs  a  math¬ 
ematical  computer  model  and  is  organized  so 
specific  high-risk  drivers  can  be  identified  for 
possible  tradeoff  analyses.  The  methodology 
draws  on  the  results  of  the  software  main¬ 
tainability,  support  resources,  and  support 
ufe  '•ycle  process  evaluations.  These  stand¬ 
alone  evaluation  techniques  provide  informa¬ 
tion  on  particular  deficiencies  and  are  not 
necessarily  presented  in  relation  to  cne 
another.  RAMSS  ties  all  the  software  sup¬ 
portability  evaluation  factors  together  and 
identifies  potential  shortcomings.  Whether 
contractor  or  military  software  support  is 
planned,  RAMSS  examines  evaluation  results 
together  with  the  estimated  software  change 
workload  for  a  system  once  it  transitions 
from  development  to  its  operational  state. 
Findings  are  presented  in  terms  of  risk  to 
the  using  and  supporting  commands. 

(1)  Method.  The  DSE  or  software  evalu¬ 
ation  manager  uses  the  guidelines  contained 
in  draft  AFOTECP  800-2,  volume  7,  Risk 
Assessment  Methodology  for  Software  Sup¬ 
portability,  to  collect  system  workload  infor¬ 
mation  and  the  results  of  other  software 
supportability  evaluations.  This  information 
is  used  as  input  to  the  RAMSS  model  which 
calculates  risk  percentages  and  identifies 
particular  areas  of  risk. 

( 2 )  Data  Requirements.  Input  require¬ 
ments  for  RAMSS  are  (a)  the  history  of 
software  support  maintenance  activities  from 
the  RAMSS  data  base,  (b)  the  estimate  of 
user/supporter  software  support  require¬ 


ments,  and  (b)  the  results  of  the  software 
supportability  evaluations:  software  main¬ 
tainability,  software  support  life  cycle  proc¬ 
ess,  and  software  support  resources. 

(3)  Evaluation.  The  DSE  or  software 
evaluation  manager  uses  the  RAMSS  model 
to  calculate  the  software  support  risk  for  a 
particular  system  and  identify  specific  areas 
of  risk  to  senior  decision  makers. 

(4)  Assessment  Criteria.  There  are  no 
quantitative  evaluation  criteria  for  this 
evaluation;  however,  a  risk  factor  greater 
than  .5  indicates  an  area  of  potential  risk. 

12-5.  Other  Software  Evaluation  Meth¬ 
ods: 

a.  Spare  Computing  Capacity.  Spare 
computer  processing  time  and  memory  have 
both  system  effectiveness  and  suitability 
implications.  Computer  processing  time  and 
memory  must  meet  current  operational 
requirements  plus  provide  sufficient  comput¬ 
ing  capacity  to  allow  for  future  software 
requirements.  Spare  processor  time  can  be 
determined  by  examining  idle  processor  time 
as  a  percentage  of  total  available  time. 
Spare  memory  can  be  determined  by  review¬ 
ing  assembler/computer-generated  memory 
usage  tables  and  comparing  them  with  the 
total  memory  available.  Future  processing 
and  memory  growth  potential  may  also  be 
included  in  this  assessment. 

b.  Computer  Security.  This  area  of 
software  evaluation  is  still  being  developed 
and  is  only  stated  as  an  OT&E  objective  if 

(1)  required  to  support  operations  concerns, 

(2)  the  test  support  group  feels  a  need  to 
emphasize  software  security,  or  (3)  stated  as 
a  critical  issue.  Typical  requirements  ad¬ 
dress  data  protection  and  processing  integ¬ 
rity.  A  completed  software  security  evalua¬ 
tion  is  useful  for  reporting  OT&E  concerns, 
but  will  not  certify  a  system  to  process 
classified  data.  Details  on  conducting  a 
computer  security  evaluation  are  documented 
in  the  draft  AFOTECP  800-2,  volume  8, 
which  is  being  developed. 

c.  Software  Reliability.  Software  relia¬ 
bility  (demonstrated  and  projected)  is  an 
integral  part  of  system  demonstrated  and 
projected  reliability  computations.  The 
Software  Analysis  Division  has  developed  a 
method  to  project  the  effects  of  software  on 
system  reliability  at  maturity.  This  method 
uses  a  curve  fitting  technique  on  a  software 
maturity  curve  for  critical  software  failures 
to  extrapolate  to  system  maturity.  The 
method  also  allows  correction  factors  for 
imperfect  debugging  and  software  enhance¬ 
ments.  The  result  is  a  software  mean  time 
between  critical  failure  (MTBCF)  at  system 
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maturity.  This  result  can  then  be  combined 
with  the  hardware  MTBCF  to  obtain  a 
system  MTBCF  at  maturity.  More  detailed 
information  regarding  software  reliability  is 
described  in  the  Software  Analysis  Division’s 
Software  Reliability  Handbook. 

cL  Adaptive  Software  Suitability  As¬ 
sessments.  Adaptive  software  suitability 
assessments  are  structured  approaches  for 
addressing  software  concerns  during  early 
operational  and  operational  assessments 
(EOA/OA)  conducted  as  part  of  a  system’s 
operational  test  and  evaluation.  As  such, 
these  assessments  will  focus  on  (and  become 
part  of)  the  six  areas  of  emphasis  addressed 
during  early  operational  and  operational 
assessments:  program  schedule  and  resourc¬ 
es,  documentation,  user  requirements,  pro¬ 
grammatic  problems  inhibiting  OT&E,  spe¬ 
cial  field  activities,  and  programmatic  voids 
and  previous  testing.  This  assessment  meth¬ 
odology  is  still  in  the  formative  stages  and 
will  be  documented  in  AFOTECP  800-2, 
volume  9  (draft). 

12-6.  Software  Evaluation  Reporting. 
The  AFOTEC  800-2  series  of  evaluations  aie 
conducted  throughout  the  software  develop¬ 
ment  life  cycle.  There  are  two  types  of 
reports  published  detailing  the  results  of 
those  evaluations:  interim  reports  and  final 
reports. 

a.  Interim  Reports.  The  results  of 
individual  800-2  series  evaluations  are  con¬ 
densed  in  interim  reports  published  by  the 
DSE  or  the  software  evaluation  manager. 
While  there  is  no  specified  format  for  inter¬ 
im  reports,  several  things  should  be  kept  in 
mind  by  the  author: 

(1)  Graphs  and  charts  of  evaluation 
results  will  condense  a  large  volume  of  prose 
that  otherwise  becomes  cumbersome  to  read. 

(2)  The  interim  report  must  have  inter¬ 
nal  consistency,  e.g.,  if  maintainability  char¬ 
acteristics  have  a  well-above-threshold  nu¬ 
meric  score  of  4.6,  the  author  should  not  go 
into  extensive  detail  explaining  why  the 
software  is  not  maintainable.  The  4.6  rat¬ 
ing  is  self-evident. 

(3)  Present  the  good  and  bad.  The 
author  should  highlight  problems,  but  pre¬ 
sent  a  balanced  report.  Remember,  the 
interim  report  will  be  used  to  get  the  pro¬ 
gram  office  to  place  management  attention 
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on  the  problems  the  evaluators  would  like 
to  see  fixed  to  improve  the  quality  of  the 
software  system.  An  extremely  negative 
report  will  not  achieve  the  goal  and.  in  fact, 
may  be  ignored,  and  any  communication  with 
the  SPO  will  probably  be  shut  down.  When 
writing  an  interim  report,  the  author  must 
remember  that  the  target  audience  is  the 
system  program  office. 

(4)  Do  not  come  across  like  an  IG.  It 
will  be  counterproductive. 

(5)  Contact  AFOTEC/LG5  to  review  past 
interim  reports  for  format.  A  DSE  should 
work  closely  with  the  LG5  counterpart  on 
drafting  the  interim  report  before  having  the 
OT&E  test  director  sign  and  forward  the 
report  to  AFOTEC/TE. 

(6)  Ensure  all  emotionalism  is  removed. 
The  report  should  describe  evaluation  results 
and  conclusions/recommendations  supported 
by  those  results.  Unsubstantiated  or  inflam¬ 
matory  statements  destroy  the  credibility  of 
the  report. 

(7)  Ensure  AFOTEC/LG5  gets  a  copy  of 
the  final  version  of  the  report.  Also,  after 
a  system’s  final  OT&E  report  is  published, 
individual  interim  reports  should  be  collected 
and  attached  to  the  OT&E  Supplemental 
Data  Document  (SDD)  which  is  compiled  for 
the  archives. 

b.  Final  Reports.  The  software  sections 
of  a  system’s  OT&E  final  report  are  the 
culmination  of  a  program’s  OT&E.  These 
sections  should  be  the  best  software  OT&E 
products  written.  There  are  specific  rules 
for  writing  final  reports,  but  not  much  detail 
for  the  software  specific  sections.  AFOTEC/ 
LG5  has  published  a  "Final  Test  Report 
Handbook,  The  LG5  Handbook  to  Writing 
Software  Portions  of  OT&E  Final  Reports  ' 
(LG5  Handbook  55-43,  1  January  1991) 
which  is  updated  on  an  annual  basis  to 
capture  new  AFOTEC  reporting  policy. 
Copies  of  the  handbook  are  available  from 
LG5.  DSEs  and  report  authors  should  not 
wait  until  the  handbook  is  needed  for  a  final 
report,  but  should  get  It  as  soon  as  possible 
as  the  guidance  in  the  handbook  provides  an 
excellent  framework  for  what  the  total  focus 
of  the  software  OT&E  should  be.  Further¬ 
more,  anyone  having  suggestions  on  how  to 
improve  the  final  report  fidelity  should  not 
hesitate  to  contact  the  Division  Chief,  Soft¬ 
ware  Analysis  Division  (AFOTEC/LG5). 
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13-1.  Introduction.  This  chapter  presents 
background  material  on  developing  an  under¬ 
standing  of  integrated  diagnostics  concerns 
as  applicable  to  the  maturation  and  testing 
of  Air  Force  systems. 

a.  Purpose: 

(1)  This  is  a  guide  for  HQ  AFOTEC 
logistics  staff  members  who  evaluate  inte¬ 
grated  diagnostics  as  part  of  the  operational 
suitability  evaluation.  An  integrated  diag¬ 
nostics  concept  impacts  many  aspects  of  a 
system’s  development  and  utilization.  It 
encourages  the  incorporation  of  adequate 
diagnostics  capability  early  on  in  the  con¬ 
ceptual  and  design  phases  of  a  system  and 
broadens  the  depth  and  scope  of  the  system’s 
diagnostics  as  it  matures  through  develop¬ 
ment  and  deployment.  Proper  implementa¬ 
tion  of  integrated  diagnostics  ensures  the 
best  possible  mix  of  diagnostic  resources  will 
be  available  to  support  the  fielded  system. 

(2)  A  system’s  intended  integrated  diag¬ 
nostics  design  drives  much  of  its  logistics 
support  planning.  Maintenance  training, 
spares,  support  equipment,  and  technical 
data  are  planned  to  support  the  system’s 
requirements  for  a  comprehensive  and  effec¬ 
tive  integrated  diagnostics  system.  When 
the  diagnostics  do  not  perform  as  expected, 
one  or  more  of  these  logistics  factors  must 
be  altered  to  provide  the  necessary  system 
support.  Significant  changes  in  a  system’s 
diagnostics  capability  or  resources  are  often 
unprogrammed  and  expensive.  Delays  in 
implementation  impact  cost  and  schedule, 
and  until  workarounds  are  established, 
system  support  suffers.  Therefore,  an  opera¬ 
tional  evaluation  of  the  overall  integrated 
diagnostics  is  needed  as  early  in  the  acquisi¬ 
tion  cycle  as  possible.  This  chapter  explains 
some  of  the  aspects  of  an  integrated  diagnos¬ 
tic  evaluation  and  focuses  on  the  critical  role 
the  automated  diagnostics  plays  in  support 
of  the  overall  diagnostic  requirements. 

b.  Scope  of  the  Evaluation  Method. 

Integrated  diagnostics  usually  serve  two 
functions:  to  monitor  and  report  system 

status  for  an  operator  and  to  be  used  as  a 
maintenance  tool  for  repair. 

(1)  For  the  first  function,  the  diagnostics 
are  used  to  monitor  system  performance  and 
provide  any  necessary  indication  of  critical 
system/subsystem  performance  degradation. 
This  indication  provides  the  operator  with 
the  necessary  information  to  decide  whether 
to  rely  on  that  system/subsystem’s  perform¬ 


ance  or  to  revert  to  an  available  backup 
system.  If  no  backup  system  is  available, 
the  operator  must  decide  if  the  mission’s 
operation  should  be  aborted. 

(2)  For  the  second  function,  the  diag¬ 
nostics  are  used  to  confirm  the  initial  indica¬ 
tion  a  malfunction  exists  and  then  to  indi¬ 
cate  what  type  of  corrective  action  is  needed 
to  restore  the  system. 

(3)  The  focus  on  the  evaluation  method 
described  in  this  chapter  is  on  the  second 
function,  evaluating  how  well  a  system’s 
diagnostics  perform  as  an  O-level  mainte¬ 
nance  tool.  However,  this  method  does 
provide  some  information  on  how  well  the 
diagnostics  support  the  system  operators  and 
the  system’s  mission  reliability.  This  evalua¬ 
tion  method  applies  to  all  systems  from 
aircraft  and  missiles  to  ground-based  elec¬ 
tronic  systems.  The  methods  should  be 
tailored  to  the  specific  system  being  evalu¬ 
ated. 

c.  Definitions.  An  understanding  of 
certain  terms  is  needed  for  evaluating  inte¬ 
grated  diagnostics.  Following  are  definitions 
of  essential  terms.  Other  definitions  related 
to  diagnostics  can  be  found  in  MIL-STD 
1309C,  Definitions  of  Terms  for  Test,  Meas¬ 
urement,  and  Diagnostic  Equipment. 

(1)  Diagnostics.  The  process  employed  to 
identify  and  isolate  system  malfunctions. 

(a)  Automated  Diagnostics  (AD).  Any 
combination  of  software,  firmware,  or  hard¬ 
ware  fault  detection  techniques  that,  once 
initiated,  require  no  further  operator  inter¬ 
vention.  These  built-in  test  (BIT)  or  self¬ 
test  techniques  may  be  periodic  (continuous) 
or  have  some  finite  number  of  predetermined 
repetition?. 

(b)  Semiautomated  Diagnostics.  A  set 
of  software/firmware/hardware  diagnostic 
techniques  that  require  some  level  of  opera¬ 
tor  or  maintenance  technician  interaction. 
These  diagnostics  are  designed  to  aid  in 
identifying  and  isolating  system  malfunctions 
and  may  include  built-in  test  equipment 
(BITE)  such  as  displays  (digital/analog  wave¬ 
forms  or  alphanumeric),  procedural  switch 
actions,  or  reading  of  various  meters  or 
indicators. 

(c)  Manual  Diagnostics.  A  diagnostic 
process  that  is  based  on  the  system  opera¬ 
tors  or  maintenance  technicians  using  obser¬ 
vations  of  system  performance,  knowledge  of 
system  design  and  operation  based  on  their 
training  and  experience,  logical  analysis. 
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external  test  equipment,  and  technical  data. 

(d)  Integrated  Diagnostics.  The  process 
of  efficiently  using  the  most  effective  combi¬ 
nation  of  a  system’s  automated,  semiauto- 
mated,  and  manual  diagnostics  resources  in 
order  to  identify  and  unambiguously  isolate 
the  cause  of  malfunctions. 

(2)  Anomaly.  Any  other  than  normal 
occurrence  such  as  false  alarms,  system 
malfunctions,  or  identified  system  failures. 

(3)  Malfunction.  Any  system  perform¬ 
ance  degradation  that  may  require  corrective 
maintenance. 

(4)  Failure.  A  physical  condition  that 
causes  a  device,  component,  or  element  to 
fail  to  perform  in  a  required  manner. 

(5)  Fault  Indication.  Any  device  which 
can  be  used  to  convey  to  the  operators  an 
indication  of  system  degradation.  This  may 
include  audible  alarms,  visual  displays  gauge 
readings,  etc. 

(6)  Fault  Isolation  (FI).  The  process  of 
isolating  the  cause  of  a  fault. 

(7)  False  Alarm  (FA).  An  indication  of 
a  system  malfunction  without  sufficient 
confirmation  of  the  system’s  degradation  to 
result  in  a  requirement  for  any  corrective 
maintenance  action.  (This  definition  differs 
from  the  definition  in  MIL-STD  1309C.)  A 
key  point  to  note  is  a  false  alarm  does  not 
generate  a  maintenance  action. 

(8)  Cannot  Duplicate  (CND).  An  opera¬ 
tionally  observed/recorded  system  malfunc¬ 
tion  that  maintenance  personnel  were  unable 
to  duplicate. 

(9)  Retest  Okay  (RTOK).  A  unit  that 
has  been  identified  as  malfunctioning  at  one 
maintenance  level,  but  the  specific  malfunc¬ 
tion  cannot  be  duplicated  at  a  higher  main¬ 
tenance  level. 

(10)  Vertical  Testability.  The  inherent 
diagnostic  capability  at  each  level  of  main¬ 
tenance  to  ensure  any  associated  malfunc¬ 
tion,  identified  to  a  specific  unit  under  test 
(UTJT)  at  one  level  of  maintenance,  can  also 
be  replicated  at  any  of  the  other  mainte¬ 


nance  levels  designated  for  that  unit. 

(11)  Built-In  Test/Fault-Isolation  Test 
(BIT/FIT)  Effectiveness.  BIT/FIT  effective¬ 
ness  is  the  capability  of  system  automated 
diagnostics  to  properly  detect  critical  system 
malfunctions,  minimize  the  diagnostic  time 
for  fault-isolation  and  repair  verification, 
minimize  the  diagnostic  time  for  line-replace¬ 
able  units  (LRU)  that  are  erroneously  iden¬ 
tified  as  malfunctioning,  and  minimize  main¬ 
tenance  man-hours.  BIT/FIT  effectiveness  is 
not  evaluated  independently  but  rather  is 
evaluated  as  to  its  influence  on  quantitative 
maintainability.  BIT/FIT  effectiveness  is  a 
subjective  assessment  of  the  utility  of  BIT/ 
FIT  as  a  maintenance  tool  and  its  impact  on 
maintenance  resource  requirements.  A 
single-thread  data  system,  ensuring  an  audit 
trail,  should  be  employed  to  track  malfunc¬ 
tions,  subsequent  repair  actions,  and  post¬ 
repair  performance  of  the  alleged  faulty 
system.  The  data  system  should  possess  the 
capability  to  track  equipment  by  serial  num¬ 
ber,  part  number,  job  control  number,  work 
unit  code,  and  aircraft  tail  number.  Addi¬ 
tional  information  captured  should  include 
total  repair  time  from  start  of  the  mainte¬ 
nance  event  to  its  completion,  associated 
corrective  action,  method  of  fault  identi¬ 
fication,  and  subsequent  fault  isolation  from 
on-equipment  discovery  through  repair  of  the 
replaced  component. 
cL  Repair  Process: 

(1)  Planning  an  operational  evaluation  of 
integrated  diagnostics  requires  an  under¬ 
standing  of  the  role  of  diagnostics  in  the 
repair  process.  The  process  begins  when  a 
malfunction  is  identified  during  system 
operation  and  progresses  through  the  five 
phases  shown  in  figure  13-1. 

(a)  In  the  setup  phase  of  the  repair 
process,  access  panels  are  opened,  required 
equipment  is  hooked  up,  switches  are  set, 
power  is  applied,  and  other  necessary  pre¬ 
paratory  tasks  performed.  This  work  is  pre¬ 
dominantly  manual. 
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Figure  13-1.  Repair  Process 


AFOTECP  400-1  15  May  1991 


13-3 


(b)  The  diagnosis  or  troubleshooting 
phase  is  usually  performed  in  two  steps. 
The  fault  confirmation  step  attempts  to  con¬ 
firm  the  problem  reported  by  the  equipment 
operator  does  exist.  Once  a  fault  is  con¬ 
firmed,  the  FI  step  attempts  to  locate  the 
cause  of  fault.  These  steps  can  be  per¬ 
formed  manually,  semiautomatically,  auto¬ 
matically,  or  by  some  combination  of  the 
three  methods. 

(c)  In  the  fault-correction  phase,  proper 
operation  of  the  system  is  restored  by  ad¬ 
justment,  removing  and  replacing  broken 
components,  etc.  These  actions  are  also 
predominantly  manual. 

(d)  The  checkouts  phase  confirms  the 
fault  has  been  corrected,  usually  by  repeat¬ 
ing  the  fault  confirmation  step  of  the  diag¬ 
nosis  phase.  The  work  in  this  phase  also 
may  be  performed  manually,  semiautomat¬ 
ically,  automatically,  or  by  some  combination 
of  the  three  methods. 

(e)  The  closeout  phase  reverses  the 
work  done  in  the  setup  phase,  readying  the 
system  for  operation.  The  work  here  again 
is  predominantly  manual. 

(2)  The  personnel  involved  in  each  of  the 
five  phases  of  the  repair  process  may  require 
the  maintenance  support  elements  of  techni¬ 
cal  data,  support  equipment,  tools,  facilities, 
and  personnel  training.  In  addition,  spare 
parts  are  often  required  for  the  diagnosis 
and  fault  phases. 

(3)  The  repair  process  takes  time.  Since 
the  objective  of  the  repair  process  is  to 
restore  a  system  to  operation,  how  quickly 
the  process  can  be  performed  is  the  primary 
measure  of  the  efficiency  of  the  process. 
Repair  time  depends  not  only  on  the  type  of 
malfunction  but  also  on  the  maintainability 
designed  into  the  system  and  the  effective¬ 
ness  of  the  support  elements.  Each  of  these 
factors  can  vary  for  each  phase  of  the  repair 
process.  For  example,  an  aircraft  fuel  leak 
can  be  detected  and  isolated  in  minutes,  but 
because  of  the  design  of  the  fuel  cells  and 
the  need  for  special  tools  and  facilities,  fault 
correction  may  take  many  hours.  On  the 
other  hand,  diagnosing  the  cause  of  an 
avionics  malfunction  may  take  hours  and 
require  not  only  built-in  AD  but  also  exter¬ 
nal  test  equipment.  However,  because  of  easy 
accessibility  and  modular  component  design, 
fault  correction  may  take  only  minutes.  Both 
malfunctions  may  take  the  same  length  of 
time  to  repair,  but  the  time  required  for  each 
repair  phase  will  vary  considerably. 

e.  Diagnostics  Design: 

(1)  Evaluation  planning  requires  an 
understanding  of  general  diagnostics  design. 
Typically,  the  system  for  which  the  diagnos¬ 


tics  are  designed  is  divided  into  those  areas 
which  are  addressable  by  AD  and  those  areas 
which  are  not. 

(2)  Designers  of  AD  do  not  attempt  to 
have  their  diagnostics  detect  and  isolate  all 
faults.  Faults  that  can  be  readily  detected 
visually,  such  as  broken  knobs  and  cracked 
indicators,  are  excluded  from  their  AD  de¬ 
sign.  Although  analog  systems  are  less 
reliable  than  digital  systems,  they  have  less 
AD  incorporated  within  their  system  because 
of  cost  constraints.  Digital  systems,  while 
having  higher  reliability,  are  often  extensive¬ 
ly  integrated  with  AD  because  of  lower 
diagnostics  development  costs.  Circuits  used 
to  check  wiring  and  interface  problems  are 
costly  and  tend  to  somewhat  lower  total 
system  reliability.  Diagnostics  that  could 
detect  multiple  failure  scenarios  are  realiz¬ 
able  but  are  often  considered  cost  prohibi¬ 
tive.  Two  practical  constraints  other  than 
costs  that  often  limit  the  application  of  AD 
are  the  space  and  weight  factors.  These 
factors  present  a  more  significant  restriction 
to  airborne  AD  systems  than  ground  based 
systems. 

(3)  The  fault  detection  (FD)  specifications 
are  relevant  in  the  operational  environment 
where  the  concern  is  knowing  when  and 
what  functions  of  a  system  are  not  perform¬ 
ing  properly.  However,  the  FI  specifications 
must  be  primarily  concerned  with  unambigu¬ 
ously  isolating  the  system  faults  as  quickly 
as  possible.  Automated  FI’s  contribution  to 
expeditions  repair  depends  on  how  well  the 
automated  diagnostics  are  integrated  into  an 
interactive  and  flexible  maintenance  concept. 

(4)  Several  factors  can  degrade  the  ex¬ 
pected  effectiveness  of  AD.  The  sensitivity 
of  FD  can  be  so  great  that  momentary  tran¬ 
sients  are  displayed  or  recorded  as  faults 
when,  in  fact,  the  system  continues  to  per¬ 
form  satisfactorily.  This  condition  results  in 
a  high  number  of  FAs  or  CND  maintenance 
actions.  Conversely,  FD  can  be  so  insensi¬ 
tive  that  true  system  faults  such  as  inter¬ 
mittent  which  should  be  detected  are  not. 
Either  situation  may  result  in  loss  of  user 
confidence  in  the  AD. 

(5)  For  some  systems,  particularly  air¬ 
craft,  operating  and  maintenance  environ¬ 
ments  differ.  FD,  designed  to  operate  while 
the  aircraft  is  flying,  may  indicate  faults 
caused  by  G-forces,  vibration,  temperature 
extremes,  radio- frequency  interference  (RFI), 
or  electromagnetic  interference  (EMI).  Since 
these  in-flight  conditions  are  not  duplicated 
on  the  ground,  fault  indications  often  term¬ 
inate  in  CND  maintenance  actions. 

(6)  In  some  systems,  the  automated 
diagnostics  are  not  designed  to  fully  isolate 
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to  the  one  malfunctioning  unit.  Instead,  the 
diagnostics  isolate  to  an  "ambiguity  group," 
e.g.,  the  malfunction  is  in  one  of  three  units. 
This  condition  usually  requires  manual  FI  to 
locate  the  malfunctioning  unit  within  the 
ambiguity  group.  Depending  on  the  size  of 
the  ambiguity  group,  this  requirement  to 
manually  isolate  the  fault  may  significantly 
extend  repair  times,  reducing  the  effective¬ 
ness  of  the  AD. 

(7)  Because  AD  design  is  often  based  on 
a  predicted  failure  distribution,  the  distri¬ 
bution  actually  experienced  in  the  operation¬ 
al  environment  may  be  significantly  dif¬ 
ferent,  and  this  may  degrade  the  effective¬ 
ness  of  the  AD.  As  an  example,  despite  the 
low  theoretical  probability  of  their  occur¬ 
rence,  multiple  faults  and  wiring  failures 
may  constitute  up  to  20  percent  of  the  oper¬ 
ational  failures  within  a  system.  If  the  AD 
is  not  designed  to  detect  and  isolate  these 
types  of  conditions,  the  troubleshooting  may 
be  ineffective  and  the  required  manual  diag¬ 
nostics  may  lengthen  repair  times. 

(8)  Test  have  shown  that  ADs  often  fail 
to  live  up  to  their  expected  capabilities.  As 
a  result,  supplemental  diagnostics  techniques 
are  usually  required  despite  high  automated 
FD  and  FI  requirements. 

13-2.  Planning  the  Evaluation: 
a.  Early  Involvement: 

(1)  Planning  for  a  diagnostics  evaluation 
must  begin  early  in  the  acquisition  cycle. 
Details  of  a  system’s  diagnostics  design  and 
the  diagnostics’  development  schedules  (in¬ 
cluding  software  updates)  are  needed  for  test 
planning.  However,  this  information  usually 
is  not  readily  available  as  early  as  it  is 
needed,  since  diagnostics  design  is  often 
deferred  until  other  aspects  of  design  are 
more  firm.  During  IOT&E,  diagnostics  is 
usually  incomplete.  Despite  this  situation, 
most  of  the  necessary  planning  information 
can  be  obtained  by  reviewing  program  docu¬ 
ments  from  attending  program  meetings. 
These  activities  give  a  logistics  evaluator  the 
opportunity  to  help  focus  on  specific  methods 
and  plans  which  can  be  used  in  the  diagnos¬ 
tics  evaluation.  In  addition,  early  involve¬ 
ment  in  the  diagnostics  design  and  develop¬ 
ment  process  can  inject  lessons  learned  from 
other  programs.  The  early  critique  of  the 
system  from  its  use  during  IOT&E  can  result 
in  effective  diagnostics  being  delivered  earlier 
than  they  otherwise  would  have  been.  The 
result  is  a  more  supportable  system. 

(2)  In  terms  of  an  operational  evaluation 
of  diagnostics,  the  most  significant  informa¬ 
tion  in  program  documents  (MNS,  ORD)  is 
the  user’s  required  system  restoration  time. 


This  requirement  could  be  expressed  as  mean 
downtime  (MDT),  mean  repair  time  <MRT\ 
maximum  time  to  repair,  or  turnaround  time. 
The  diagnostics  must  contribute  to  achieving 
this  required  restoration  time.  If  the  MNS 
or  ORD  omits  such  measures.  AFOTEC 
should  recommend  they  be  added. 

(3)  The  MNS  or  ORD  should  contain  an 
integrated  diagnostics  approach  instead  of 
automated  FD  and  FI  percentages.  These 
documents  should  recognize  the  need  for  100- 
percent  integrated  diagnostics  capability 
based  on  the  most  cost-effective  mix  of  man¬ 
ual,  semiautomated,  and  automated  diagnos 
tics.  This  requirement  simply  expresses  tne 
operational  need  to  confirm  and  isolate  all 
faults  which  prevent  or  degrade  system 
operation.  If  this  requirement  is  not  already 
in  the  documents,  include  it  in  your  recom¬ 
mended  changes. 

(4)  A  second  category  of  documents  re¬ 
lates  to  the  request  for  proposal  (RFP)  and 
acquisition  contract.  The  system  specifica¬ 
tion,  the  statement  of  work  (SOW),  the 
contract  data  requirements  list  (CDRL),  and 
the  model  contract  contain  information 
needed  for  planning  an  integrated  diagnos¬ 
tics  evaluation.  At  times,  however,  these 
documents  omit  vital  information,  or  the 
information  is  not  properly  stated.  Your 
objective  in  reviewing  the  documents  should 
be  to  obtain  necessary  test  planning  informa¬ 
tion  and  to  make  specific  recommendations 
to  correct  errors  as  early  as  possible.  Drafts 
of  these  documents  are  often  received  for 
review  individually  over  a  period  of  time. 
Requirements  stated  in  one  of  these  docu¬ 
ments  should  track  through  all  the  docu¬ 
ments,  since  each  document  should  eventual¬ 
ly  become  part  of  the  RFP/contract. 

(a)  The  specification  describes  how  the 
system  is  to  work  and  includes  required 
reliability  and  maintainability  characteristics 
as  well  as  the  overall  maintenance  concept. 
The  specification  should  describe  the  desired 
diagnostic  techniques  for  system  operation 
and  maintenance.  It  should  state  whether 
AD  will  continuously  monitor  system  per¬ 
formance  or  be  initiated  by  the  operator  and 
whether  these  diagnostic  results  will  be 
recorded  for  later  maintenance  analysis.  It 
should  define  the  allowable  FA  and  CND 
parameters  and  the  size  of  the  FI  ambiguity 
group.  The  specification  should  describe  the 
relationship  of  AD  to  the  various  subsystems 
as  well  as  to  external  support  equipment.  It 
should  include  the  required  system  restora¬ 
tion  time  from  the  MNS  or  ORD  and  de¬ 
scribe  how  the  diagnostics  will  support  meet¬ 
ing  this  requirement.  The  specification 
should  indicate  the  areas  of  the  design  that 
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will  be  addressable  by  AD  and  what  areas 
will  be  solely  dependent  on  manual  diagnos¬ 
tics.  The  specification  may  include  auto¬ 
mated  FD  and  FI  percentages,  but  these 
figures  can  be  misleading.  They  should  sho%v 
a  direct  relationship  to  the  achievement  of 
100-percent  critical  FD  and  100-percent  FI 
through  a  combination  of  automated,  semi- 
automated,  and  manual  diagnostics. 

(b)  The  SOW  describes  the  actions  the 
contractor  must  take  to  produce  an  item  that 
will  meet  the  specification.  The  SOW  should 
include  a  section  on  integrated  diagnostics. 
This  section  should  require  the  contractor  to 
prepare  an  integrated  diagnostics  plan,  to 
perform  design  and  analysis,  and  to  conduct 
tests  to  ensure  a  100-percent  integrated 
diagnostics  capability  is  provided  by  the  most 
effective  mix  of  BIT,  support  equipment, 
technical  data,  and  training  to  achieve  stated 
operational  requirements.  The  SOW  should 
also  require  the  contractor  to  ensure  vertical 
testability  among  all  levels  of  necessary 
contractor  support  during  OT&E.  When  the 
contractor  must  maintain  the  system  during 
test,  the  SOW  should  require  the  contractor 
to  perform  maintenance  using  the  same 
diagnostic  techniques  that  will  be  provided 
for  Air  Force  use  in  the  operational  environ¬ 
ment.  The  SOW  should  also  identify  wheth¬ 
er  the  contractor  or  blue-suiters  will  docu¬ 
ment  maintenance  actions  and  specify  a 
maintenance  data  collection  system  accept¬ 
able  to  both  the  DT&E  and  OT&E  com¬ 
mands. 

(c)  The  CDRL  identifies  all  the  docu¬ 
ments  the  government  intends  to  buy  from 
the  contractor.  Each  document  is  described 
on  a  DD  Form  1423,  CDRL.  CDRL  is  a 
package  of  these  forms  and  is  created  from 
an  earlier  action  called  the  "data  call''  where 
interested  government  agencies  can  identify 
the  contractor  data  they  need.  Actions 
described  in  the  SOW  should  have  cor¬ 
responding  documents  listed  in  the  CDRL. 
Any  CDRL  document  must  have  a  cor¬ 
responding  SOW  action.  Contractor  docu¬ 
ments  relevant  to  planning  an  integrated 
diagnostics  evaluation  are  listed  in  figure 
13-2.  This  list  should  be  checked  for  cur¬ 
rency  before  requesting  any  documents. 
During  the  data  call  and  CDRL  review, 
ensure  these  documents  are  listed  when 
appropriate  and  your  office  is  on  distribution 
for  them. 

(d)  The  model  contract  is  a  draft  of  the 
contract  intended  for  the  winner  of  the 
source  selection.  It  should  contain  the  key 
requirements  from  the  specification,  SOW, 
and  CDRL.  Any  changes  you  recommend  in 
these  documents  should  also  be  added  to  the 


model  contract. 

(5)  For  source  selecting,  contractors  are 
able  to  propose  exceptions  to  the  model 
contract.  If  you  participate  in  the  source 
selection,  study  the  contractors'  proposed 
exceptions  carefully.  Regardless  of  what  is 
stated  in  other  sections  of  the  proposal,  what 
the  contractors  really  intend  to  deliver  is 
expressed  in  their  proposed  model  contract. 
For  example,  the  SOW  requirement  to  en¬ 
sure  100-percent  integrated  diagnostics  capa¬ 
bility  may  be  proposed  for  deletion  in  a 
contractor’s  model  contract  despite  other 
parts  of  the  proposal  supporting  it. 

(6)  After  source  selection,  carefully  re¬ 
view  the  actual  contract  to  ensure  the  diag¬ 
nostics  provisions  from  the  specification, 
SOW,  CDRL,  and  model  contract  survived 
contract  negotiation.  What  is  in  the  contract 
has  a  significant  influence  in  test  planning. 
If  it  is  not  clearly  identified  within  the 
contract,  do  not  expect  to  see  it  in  the  sys¬ 
tem. 

(7)  Much  of  the  additional  information 
needed  for  evaluation  planning  is  obtainable 
at  various  system  program  meetings.  In 
addition  to  providing  planning  information, 
these  meetings  also  allow  the  logistics  evalu¬ 
ator  the  opportunity  to  influence  integrated 
diagnostics  development. 

(8)  Integrated  logistics  support  and  other 
logistics-oriented  meetings  can  provide  such 
information  as  delivery  schedules  for  techni¬ 
cal  data  and  support  equipment.  These 
meetings  are  usually  attended  by  key  system 
program  managers  and  their  staffs  who  are 
responsible  for  the  design  of  the  automated 
diagnostics.  These  logistics  managers  usual¬ 
ly  assume  in  their  logistics  planning  the 
diagnostics  will  be  effective.  Any  shortfalls 
in  the  expected  AD  design  will  impact  their 
logistics  planning.  The  logistics  managers 
can  provide  the  necessary  program  redirec¬ 
tion  to  head  off  anticipated  shortfalls  only  if 
time  and  money  are  available.  For  this 
reason,  early  information  on  potential  short¬ 
falls  is  essential.  Through  early  involvement 
in  the  program,  the  AFOTEC  logistics  eval¬ 
uator  is  often  able  to  provide  this  infor¬ 
mation. 

(9)  Other  needed  information  can  be 
obtained  from  design  reviews  and  similar 
engineering  meetings.  These  meetings  are 
attended  by  engineers  responsible  for  the 
design  and  development  of  a  system’s  diag¬ 
nostics  capability.  These  people  are  able  to 
provide  detailed  information  on  diagnostics 
design,  but  are  less  aware  of  the  impact 
their  design  will  have  on  the  maintenance 
support  elements  being  planned.  Again,  the 
AFOTEC  logistics  evaluator  is  often  in  a 
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Data  Item 
Description 
(DID)  Number 

US-92-AFLD  and 
UL-780-ESD 


DI-T-3734A/M 

DI-E-3120A 

DI-A-6102A/M 

DI-L-6138 

DI-R-3533 

DI-T-3701 

DI-A-3009 

DI-A-6101A/M 


DI-E-3101 


DI-H-6131M 


Title 

Integrated  Diagnostics  Plan 


Test  Requirements  Document 
(TRD) 

Computer  Program  Product  Spec¬ 
ification 

Support  Equipment  Plan 

Integrated  Support  Plan 

Reliability/Maintainability  Pro¬ 
gram  Plan 

System  Test  Plan 

Program  Milestones 

Contractor  Engineering  and  Tech¬ 
nical  Services  Plan 


System  Specification 


Training  and  Training  Equip¬ 
ment  Plan 


Description 

Outlines  contractor’s  program  for 
achieving  1009b  fault  detection  and 
isolation.  These  two  DIDs  were 
created  for  the  HH-60D  and  North 
Warning  System  (NWS)  line  radar 
replacement  programs  respecifica¬ 
tions  but  could  be  applied  to  other 
programs. 

Describes  line-replaceable  unit 
(LRU)  test  conditions. 

Describes  software  organization. 
Includes  flowcharts  and  listings. 

Describes  support  equipment. 
Useful  for  determining  what  tests 
require  support  equipment  to  sup¬ 
plement  automated  diagnostics. 

Comprehensive  plan  for  integrated 
logistics  support.  May  contain  the 
integrated  diagnostics  plan. 

Maintainability  portion  useful  for 
determining  contractor  develop¬ 
ment  planning  for  diagnostics. 

Provides  3cope  of  contractor's  test 
program.  May  include  demonstra¬ 
tion  of  diagnostics. 

Provides  current  schedule  for  deliv¬ 
ery  of  other  data  items,  including 
those  supporting  integrated  diag¬ 
nostics. 

Spells  out  the  level  of  contractor 
(vs  Air  Force)  involvement  in 
maintenance  during  test.  'Will  Air 
Force  technicians  be  using  inte¬ 
grated  diagnostics  to  maintain  the 
test  system  or  will  contractor 
people  do  it  with  workarounds?" 

Describes  the  total  system  in  de¬ 
tail.  Should  cover  what  the  con¬ 
tractor  plans  to  provide  in  the  way 
of  integrated  diagnostics.  (This  is 
not  the  same  system  specification 
from  the  RFPj 

Shows  how  the  contractor  plans  to 
handle  the  training  end  of  the 
integrated  diagnostics. 


Figure  13-2.  Contractor  Data  Needed  for  Diagnostics  Evaluation  Planning 
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Data  Item 

Description 
(DID)  Number 

Title 

Description 

DI-S-3606 

System/Design  Trade  Study  Re¬ 

Can  be  used  to  document  how 

ports 

tradeoffs  will  be  performed  among 
automated,  semiautomated,  and 
manual  diagnostics  techniques. 

Figure  13-2  (continued) 

position  to  point  out  this  impact. 

(10)  Other  program  meetings  are  ori¬ 
ented  to  the  system's  software.  These  meet¬ 
ings  can  provide  diagnostics  software  design 
information  and  delivery  schedules.  This 
information  is  essential  for  logistics  evalua¬ 
tion  planning  since  software  is  usually  devel¬ 
oped  and  delivered  incrementally.  For  this 
reason,  full  diagnostics  capability  is  often  not 
achieved  until  late  in  the  acquisition  sched¬ 
ule.  The  operational  evaluation  should  be 
planned  to  look  at  each  successive  software 
update.  OT&E  results  can  provide  the 
necessary  feedback  to  correct  software  errors 
in  subsequent  updates. 

(11)  In  some  programs,  where  system 
support  depends  heavily  on  AD,  a  diagnos¬ 
tics  development  working  group  may  be 
formed.  Meetings  of  these  groups  provide 
vital  information  on  all  aspects  of  the  diag¬ 
nostics.  The  involvement  of  the  logistics 
evaluator  in  such  groups  is  essential. 

(12)  The  meetings  mentioned  above  do 
not  automatically  provide  the  information 
needed  for  planning  a  diagnostics  evaluation. 
The  information  usually  comes  only  by  ask¬ 
ing  the  right  questions.  Answers  to  ques¬ 
tions  like  the  following  will  provide  informa¬ 
tion  on  what  we  need  to  be  aware  of  and 
will  guide  the  scope  of  the  evaluation: 

(a)  What  are  the  differences  in  proc¬ 
essing  logic  and  fault  display  between  diag¬ 
nostics  for  system  operators  and  diagnostics 
for  maintenance  technicians? 

(b)  Does  the  AD  have  more  than  one 
mode  to  accommodate  different  operational 
maintenance  conditions?  Examples:  Proce¬ 
dures  may  differ  from  main  operating  base 
(MOB)  to  forward  operating  bases,  diagnos¬ 
tics  may  be  designed  to  work  around  por¬ 
tions  of  the  system  that  may  receive  battle 
damage,  or  diagnostics  can  be  selective  under 
certain  operational  conditions  to  expedite 
faster  turnarounds  using  larger  ambiguity 
groups,  combined  with  a  liberal  LRU  replace¬ 
ment  policy  (wholesale  ambiguity  group 


replacement). 

(c)  What  techniques  are  provided  to 
automatically  record  malfunctions? 

(d)  What  equipment  is  available/re¬ 
quired  to  interrogate  recorded  diagnostics 
data  for  subsequent  troubleshooting? 

(e)  What  techniques  are  available 
during  maintenance  to  duplicate  various 
malfunctions  that  can  result  from  a  combina¬ 
tion  of  environmental  conditions  that  result 
directly  from  operationally  induced  stresses? 
(This  is  particularly  significant  for  aircraft 
systems.) 

(f)  What  techniques  are  used  to  mini¬ 
mize  operator  errors? 

(g)  What  techniques  are  used  to  mini¬ 
mize  FAs? 

(h)  What  techniques  are  used  to  mini¬ 
mize  CNDs  and  RTOKs? 

(i)  Can  the  automated  diagnostics 
detect  any  gradual  system  degradation  or 
intermittent  failures,  and  can  it  separate 
these  types  of  significant,  but  hard  to  detect, 
faults  from  fault  indications  caused  by  tran¬ 
sients  that  often  result  in  FA? 

(j)  What  techniques  are  available  to 
the  operator  to  confirm,  reset,  or  update  a 
system’s  operational  status? 

(k)  What  is  the  LRU  ambiguity  level  of 
FI? 

(l)  Do  FI  procedures  take  into  account 
wiring  and  interface  failures? 

(m)  To  what  failure  level  have  the 
diagnostics  been  designed  (internal  to  chip, 
input/output  (I/O)  of  chips,  internal  to  card, 
I/O  of  cards,  within  the  LRU,  I/O  of  LRU,  at 
a  higher  level)? 

(n)  To  what  sensitivity  level  have  the 
diagnostics  been  designed  (less  than  worst- 
case  tolerances,  worst-case  tolerances,  beyond 
worst-case  tolerances,  for  specified  time 
intervals,  etc.)? 

(o)  Has  the  system  been  designed  to 
worst-case  combinations  of  power  or  voltage 
variations,  coupled  with  environments  lly  in¬ 
duced  degradations  (i.e.,  temperature,  vibra- 
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tion,  humanity,  etc.)? 

(p)  Have  the  same  worst-case  design 
techniques  used  on  the  system  hardware 
been  used  for  diagnostics  hardware  design? 

(q)  Has  adequate  time  been  provided  to 
validate  system  operation  after  the  DT&E 
hardware,  firmware,  and  software  changes 
have  been  incorporated  within  the  system 
prior  to  entering  IOT&E? 

(r)  What  is  the  schedule  for  software 
updates  that  contain  changes  to  the  diagnos¬ 
tics? 

(s)  What  diagnostics  improvements  will 
result  from  these  updates? 

(t)  What  provisions  have  been  made  for 
vertical  testability? 

(u)  What  techniques  are  used  to  detect 
and  isolate  multiple  faults? 

(v)  What  provisions  have  been  made  to 
ensure  the  availability,  reliability,  and  main¬ 
tainability  of  diagnostic  support  elements 
such  as  test  equipment  (TE)  and  technical 
orders  (TO)? 

b.  Scoping  the  Evaluation: 

(1)  Once  an  understanding  of  the  diag¬ 
nostic  parameters  has  been  obtained,  the 
next  step  is  to  scope  the  diagnostics  evalua¬ 
tion  to  prepare  for  writing  the  test  plan.  An 
operational  evaluation  of  integrated  diagnos¬ 
tics  involves  much  effort,  detailed  data  col¬ 
lection,  and  dedicated  test  team  resources. 
To  provide  a  good  evaluation  and  not  waste 
resources  requires  careful  scoping. 

(2)  Several  factors  drive  the  scope  of  an 
evaluation.  Consider  the  following: 

(a)  Significance  of  the  diagnostics  to 
the  maintenance  concept.  System  support, 
which  relies  heavily  on  BIT  integration 
throughout  the  major  subsystems,  warrants 
an  in-depth  diagnostics  evaluation.  On  the 
other  hand,  an  evaluation  of  a  simple  go/no- 
go  test  routine  for  a  noncritical  subsystem 
should  be  incorporated  in  an  overall  main¬ 
tainability  evaluation. 

(b)  Maturity  of  the  diagnostics  during 
operational  testing.  Diagnostics  development 
usually  lags  the  rest  of  system  design.  Early 
operational  testing  must  contend  with  im¬ 
mature  diagnostics.  Diagnostics  software, 
TOs,  and  support  equipment  are  delivered 
incrementally,  often  at  the  end  of  IOT&E. 
Do  not  plan  an  extensive  evaluation  of  diag¬ 
nostics  if  the  full  diagnostics  capability  is 
not  scheduled  to  be  delivered  until  after  the 
test  period. 

(c)  Availability  of  the  appropriate  per¬ 
sonnel  for  the  test  team.  In  addition  to 
equipment  operators  and  maintenance  tech¬ 
nicians  who  use  the  diagnostics  and  generate 
data,  the  scope  of  an  operational  evaluation 
may  be  severely  limited  if  the  following  types 


of  personnel  are  not  available. 

1.  Data  managers  to  organize  and 
control  the  data  collection  process. 

2.  Additional  maintenance  technicians 
dedicated  to  guiding  data  collection  and 
evaluating  the  data. 

(d)  Intended  length  of  the  test  period 
and  the  extent  to  which  the  diagnostics  will 
be  used  during  this  period.  Money  and 
schedules  limit  the  time  available  for  opera¬ 
tional  evaluation.  A  thorough  operational 
evaluation  of  a  complex  system  may  require 
a  minimum  of  4  to  6  months  of  daily  use  of 
the  diagnostics.  Even  if  this  time  is  avail¬ 
able,  a  highly  reliable  system  or  a  system 
operated  infrequently  will  limit  the  use  of 
diagnostics  and,  consequently,  the  evaluation. 

(e)  Availability  of  both  a  user-approved 
maintenance  concept  and  an  adequate  main¬ 
tenance  data  collection  system  during  test. 
When  a  system  is  maintained  during  OT&E 
by  a  contractor  who  provides  the  mainte¬ 
nance  data,  as  sometimes  happens  during 
combined  development  and  operational  test¬ 
ing,  the  chances  are  slim  the  maintenance 
data  will  be  detailed  enough  to  support  a 
diagnostics  evaluation.  In  this  case,  you 
should  plan  on  extra  test  team  members  to 
monitor  the  contractor’s  use  of  diagnostics 
and  to  document  the  troubleshooting  times 
and  other  maintenance  data  in  an  Air  Force 
data  collection  system.  Video  recording 
equipment  and  other  means  of  diagnostics 
data  collection  should  be  considered. 

(f)  Whether  or  not  the  automated 
diagnostics  has  a  built-in  recorder  that  will 
allow  analysis  of  diagnostics  data  generated 
during  system  operation,  coupled  with  a 
printing  and  interpretation  capability,  such 
a  recorder  can  capture  data  that  could  be 
especially  useful  in  determining  FD  capa¬ 
bility  and  the  extent  and  nature  of  FAs  and 
CNDs.  However,  qualified  people  and  time 
are  needed  to  evaluate  these  data. 

These  factors  need  to  be  weighed  in  deter¬ 
mining  the  overall  scope  of  a  diagnostics 
evaluation.  Weighing  these  factors  is  "coarse 
tuning"  the  evaluation.  "Fine  tuning"  comes 
during  detailed  planning. 

c.  Detailed  Planning:  Detailed  plan¬ 
ning  includes  defining  the  objectives  and 
selecting  the  measures  to  be  incorporated  in 
the  test  plan.  For  an  evaluation  of  diagnos¬ 
tics,  plan  to  incorporate  simple,  operation¬ 
ally  relevant  objectives  and  measures  that 
can  be  derived  from  operator  and  main¬ 
tenance  data. 

(1)  The  first  step  in  detailed  planning 

should  be  obvious - thinking  clearly  about 

and  writing  down  the  objective.  Depending 
on  the  scope  of  the  evaluation,  diagnostics 
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may  be  a  separate  objective  or  embedded 
with  MOEs  under  other  objectives  such  as 
mission  reliability  or  maintainability.  In 
either  case,  the  objective  or  MOEs  are  mu¬ 
tually  dependent  on  several  other  objectives 
in  the  test  plan.  In  addition  to  maintainabil¬ 
ity  and  mission  reliability,  some  diagnostics 
MOEs  may  fall  under  logistics  supportability 
such  as  RTOKs,  technical  data,  support 
equipment,  maintenance  training,  and  soft¬ 
ware.  Each  of  these  objectives  and  support 
elements  is  interdependent  and  is  required 
to  effectively  evaluate  a  system’s  integrated 
diagnostics  capability.  However,  an  opera¬ 
tional  test  plan,  containing  multiple  objec¬ 
tives  relating  to  diagnostics,  can  result  in 
fragmenting  the  evaluation  rather  than 
guiding  the  test  team  through  an  integrated 
analysis  of  the  various  diagnostics  elements. 
The  best  way  to  state  the  objective  in  the 
test  plan  is  "evaluate  the  system’s  integrated 
diagnostics."  The  section  on  "Method"  under 
this  objective  should  detail  what  will  be 
evaluated  (hardware,  software,  displays,  etc.) 
and  describe  relationships  with  other  objec¬ 
tives  in  the  test  plan  (system  availability, 
maintainability,  and  mission  reliability  in 
addition  to  technical  data,  support  equip¬ 
ment,  training,  and  software). 

(2)  Once  you  have  structured  the  inte¬ 
grated  diagnostics  objective  within  the  over¬ 
all  test  plan,  you  are  ready  for  the  second 
step,  selecting  the  measures  that  will  help 
guide  the  evaluation.  There  are  many  meas¬ 
ures  available  for  evaluating  diagnostics. 

(a)  The  primary  approach  to  selecting 
parameters  for  an  operational  evaluation  is 
to  use  diagnostics  measures  (and  related 
criteria)  from  the  ORD. 

(b)  The  parameters  should  be  based  on 
operational  uses  of  any  integrated  diagnos¬ 
tics  assets  by  either  system  operators  or 
maintenance  technicians.  For  the  operators, 
the  diagnostics  are  supposed  to  detect  criti¬ 
cal  failures  and  convey  that  information  to 
them  in  a  clear  and  timely  manner.  The 
operator  wants  to  be  confident  that  all  criti¬ 
cal  failures  are  detected  and  any  highly 
visible  indicated  failure  represents  an  unac¬ 
ceptable  level  of  system  degradation.  The 
maintenance  technician,  on  the  other  hand, 
wants  to  be  able  to  use  integrated  diagnos¬ 
tics  to  help  confirm  a  reported  failure  and  to 
accurately,  efficiently,  and  effectively  isolate 
the  cause  of  the  failure. 

(3)  Five  measures  are  needed  to  tell  how 
well  a  system’s  integrated  diagnostics  per¬ 
form  these  operational  functions.  Each 
measure  is  made  up  of  several  data  elements 
characteristics,  and  some  measures  are 
computed  in  more  than  one  way.  These 


measures  are: 

(a)  Percentage  of  critical  faults  indi¬ 
cated  (CFI)  to  the  operators  in  a  clear  and 
timely  manner,  which  is  computed  from 
operational  and  maintenance  data  by  the 
equation: 


%  of  CFI  = 


#  of  CFI  to  operator  in  a 
timely  manner  that  resulted  in 
a  request  for  corrective 
maintenance  action  'CMA> 


#  of  CFI  to 
operator  in  + 
a  timely 
manner  that 
resulted  in 
a  CMA  request 


#  of  critical 
faults 
identified 
after-the-fact 


1.  CFIs  to  the  operator  in  a  timely 
fashion  are  any  malfunctions  that  unac¬ 
ceptably  degrade  the  system's  capability  to 
support  its  mission  requirements.  Any 
degradation  or  significant  loss  of  system 
capability  must  be  clearly  identified  to  the 
operator  in  time  for  the  information  to  be 
properly  used.  Examples  of  these  types  of 
losses  are: 

a.  A  countermeasure  system  that  is 
not  adequately  jamming  hostile  sensors,  but 
clearly  identifies  the  malfunction  to  the 
operator  in  a  timely  manner. 

b.  A  malfunction  within  a  missile 
system  that  prohibits  its  launch  or  the  mis¬ 
sile’s  capability  to  properly  acquire/lock  on 
to  a  target,  but  clearly  notifies  the  operator 
of  the  failure  within  seconds. 

c.  An  inertial  navigation  system 
that  clearly  indicates  to  the  operators  as 
soon  as  it  has  degraded  below  the  accuracy 
required  to  ensure  reasonable  mission  suc¬ 
cess. 

2.  Critical  faults  identified  after  the 
fact  fall  into  two  general  groups: 

a.  Faults  that  were  identified  to 
the  operators  at  some  time  during  the  mis¬ 
sion  but  had  not  been  immediately  identified 
at  the  time  of  their  occurrence.  The  late 
notification  did  not  permit  the  operator 
adequate  time  to  take  the  necessary  correc¬ 
tive  measures  and  resulted  in  compromising 
mission  capability.  If  a  timely  notification  of 
the  system  malfunction  can  be  reported  to 
the  operators,  it  would  permit  the  operators 
to  reduce  unnecessary  manpower/equipment 
risk  by  making  an  early-on  decision  to  abort 
the  mission,  revise  the  mission,  or  use  some 
operationally  available  backup  system/sub¬ 
system/unit.  The  system/subsystem/unit 
could  then  have  provided  the  essential  capa¬ 
bility  to  support  the  operational  requirement 
and  continue  the  mission.  Examples: 

(1)  A  missile  that  cannot  be  prop- 
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erly  launched  or,  if  the  missile  can  be  prop¬ 
erly  launched,  will  not  have  the  capability  to 
acquire/track  or  disable  its  assigned  target; 
but  unfortunately,  the  operator  was  not 
informed  of  the  system  malfunction  until  the 
actual  attempt  to  launch  the  missile. 

(2)  An  INS  that  has  degraded 
below  acceptable  mission  requirements,  but 
the  loss  of  effectiveness  was  not  promptly 
conveyed  to  the  operator.  Because  the  oper¬ 
ator  was  not  given  a  timely  fault  indication, 
no  attempt  is  made  to  utilize  any  effective 
alternate  backup  system  such  as  a  doppler 
radar. 

b.  Faults  discovered  after  the  mis¬ 
sion  has  been  completed.  These  faults  often 
are  not  identified  until  mission  performance 
data  are  reassessed. 

(1)  A  missile  that  has  a  defective 
guidance  flight  but  was  not  identified  to  the 
operator  prior  to  launch.  The  fault  is  identi¬ 
fied  after  launch  by  tracking  the  missile’s 
performance. 

(2)  Engine  foreign  object  damage 
(FOD)  during  flight,  but  only  discovered 
during  postflight  inspection. 

(b)  Unconfirmed  faults  (FA  and  CND) 
by  an  operational  unit  (UF/OU)  that  are  com¬ 
puted  from  operational  and  maintenance  data 
by  the  equation: 


UF/OU  s _ —  —  ft  °f  CND _ 

#  of  operational  units  (sorties, 
operational  hours,  etc.) 


1.  Unconfirmed  faults  are  the  sum  of 
FAs  and  CNDs  reported  to  system  operators. 

2.  The  operational  unit  of  operation 
can  be  sorties,  flying  hours,  mission  opera¬ 
tional  hours,  or  hours  of  operation  with 
power  applied. 

3.  If  appropriate,  separate  parameters 
such  as  FA/OU  and  CND/OU  may  be  used. 

(c)  Mean  time  to  troubleshoot  (MITT), 
which  is  computed  from  maintenance  data 
by  the  equation: 


mean  time  for  manual 
fauit  isolation  (MFI)  = 


total  O-level  time  spent  in 
isolating  faults  in  which 
only  manual  diagnostics 

_ were  effective _ 

#  of  O-level  FIs  in  which 
only  manual  diagnostics 
were  effective 


b.  Percentage  of  O-level  manual  FIs 
(percentage  MFI). 


%  MFI  = 


#  of  O-level  FIs  in  whichamly 
_  manual  diagnostics  were  enectfi 


#  of  O-level  troubleshooting 
actions 


I£X  100 


2.  Second  Component: 

a.  MTTT  when  BIT  contributed  to 
the  FI  process. 

total  O-level  time  spent  in  isolating 
MTTT  when  faults  in  which  BIT  contributed  to 
BIT  aids  FI  =  the  fault  isolation  process 

#  of  O-level  FIs  in  which  BIT 
contributed  to  the  FI  process 

b.  Percentage  of  BIT  contributed 
FIs  (percentage  BCFI). 


#  of  O-level  FIs  in  which 
%  BCFI  =  BIT  contributed  to  the  isolation  x  ^00 
#  of  O-level  troubleshooting 
actions 

3.  Third  Component: 
a.  Mean  time  devoted  to  investi¬ 
gating  CNDs. 

total  O-level  time  spent 
mean  time  to  investigating  malfunctions 
investigate  CND  =  -^*U«™inate  m  a  CND 
#  of  O-level  FIs  that 
terminate  in  a  CND 


b.  Percentage  of  faults  that  termi¬ 
nate  in  a  CND  (percentage  of  CND). 


MTTT  a  total  of  all  O-level  troubleshooting  times 
#  of  O-level  troubleshooting  actions 


#  of  0-leveJ  FIs_.t_ 

%  of  CND  =  that  terminate  in  a  CND  x  10q 
#  of  O-level  troubleshooting 
actions 


This  general  equation  should  be  broken  down 
into  its  three  significant  components  as 
follows: 

1.  First  Component: 
a.  MTTT  when  only  manual 
troubleshooting  techniques  are  effective. 


(1)  The  O-level  troubleshooting 
time  starts  after  setting  up  equipment  and 
turning  on  the  power  and  is  terminated  prior 
to  the  final  repair/replacement  action.  If 
units  (LRU/SRU)  are  replaced  in  order  to 
reduce  the  ambiguity  group,  the  replacement 
is  included  within  the  troubleshooting  time. 
The  troubleshooting  time  will  terminate  at 
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the  initiation  of  the  final  repair  or  unit 
replacement  action  that  unambiguously  iso¬ 
lates  the  problem. 

(2)  Only  the  O-level  CMA  that 
involved  troubleshooting  is  included  within 
the  CM As  assessed  within  the  third  measure 
of  (MTTT). 

(d)  Contribution  of  integrated  diagnos¬ 
tics  support  elements  in  troubleshooting  (i.e., 
TOs  and  TE),  which  is  assessed  from  quali¬ 
tative  maintenance  data  and  recorded  on 
forms  such  as  the  AFTO  Form  349  or  AFSC 
Form  258.  This  measure  most  fully  incor¬ 
porates  the  integrated  diagnostics  concept  by 
assessing  the  key  supporting  factors  needed 
to  ensure  an  adequate  integrated  diagnostics 
approach  has  been  implemented  in  support 
of  a  system’s  mission  requirement.  The 
principal  integrated  diagnostics  supporting 
factors  are  the  TOs,  TE,  and  training.  These 
factors  must  be  assessed  from  two  aspects: 
First,  horn  a  standpoint  of  supporting  the 
total  mission's  diagnostics  requirements;  and 
second,  from  the  aspect  of  efficiently  using 
available  personal  and  material  resources  and 
the  effectiveness  in  using  these  resources  to 
support  the  implementation  of  an  integrated 
diagnostics  concept  within  a  system. 

(e)  Percentage  of  LRU/SRU  that  retest 
okay  (RTOK)  at  a  higher  maintenance  level, 
which  is  computed  from  maintenance  data 
by  the  equation: 


#  of  LRU/SRU 
that  RTOK  at  a 

%  of  RTOK  =  maintenance  level. 

#  of  LRUs  tested  at  higher 
maintenance  level 


x  100 


1.  The  LRU/SRUs  RTOK  at  the 
higher  maintenance  level  if  they  do  not 
demonstrate  the  same  functional  anomaly 
they  demonstrated  at  the  lower  maintenance 
level. 

2.  If  the  initial  LRU/SRU  anomaly 
that  had  been  detected  and  identified  at  a 
lower  maintenance  level  can  be  reverified  as 
malfunctioning  in  the  same  manner  at  a 
higher  maintenance  level,  then  that  LRU/ 
SRU  did  not  RTOK 

Note:  Data  needed  to  assess  this  measure 
are  often  limited  until  the  later  stages  of 
operational  testing. 

13-3.  Conducting  the  Evaluation  and 
Reporting  the  Results: 
a.  Introduction: 

(1)  Conducting  the  evaluation  is  the  most 
difficult  phase  of  the  test  cycle.  The  dynam¬ 
ics  of  operational  testing  make  it  extremely 
difficult  to  control  the  data  collection  re¬ 
quired  for  a  diagnostics  evaluation.  Be 


aware  of  these  difficulties. 

<  2)  The  types  of  difficulties  vary  with  the 
type  of  operational  test.  In  a  combined 
DT&E/IOT&E,  diagnostics  is  usually  imma¬ 
ture.  This  condition  is  often  compounded  if 
the  system  is  being  maintained  by  a  con¬ 
tractor.  During  a  dedicated  IOT&E,  diagnos¬ 
tics  is  still  often  immature  resulting  in  such 
factors  as  the  operator’s  seeing  frequent  FAs 
and  reluctance  of  maintenance  to  use  the 
designated  diagnostics  for  maintenance  trou¬ 
bleshooting.  Diagnostics  may  be  more  ma¬ 
ture  during  a  follow-on  operational  test  and 
evaluation  (FOT&E).  However,  during  this 
period,  the  using  command  is  usually  oper¬ 
ating  and  maintaining  the  system  in  order  to 
achieve  an  initial  operational  capability.  The 
emphasis  is  on  rapid  maintenance  to  keep 
the  system  working. 

(3)  Other  factors  further  compound  the 
difficulty  of  conducting  diagnostics  evalua¬ 
tions.  Most  acquisition  programs  have  soft¬ 
ware  and  hardware  updates  that  are  sched¬ 
uled  to  occur  throughout  the  operational  test 
period.  Often  these  updates  are  intended  to 
improve  diagnostics  as  well  as  performance. 
If  the  originally  delivered  diagnostics  do  not 
work  well,  there  is  a  tendency  to  defer 
diagnostics  use  until  these  improvements  are 
incorporated.  To  make  matters  worse,  these 
update  schedules  frequently  slip  and  the 
improvements  are  not  incorporated  until  late 
in  the  operational  test  period.  Moreover,  in 
the  rush  to  improve  performance,  updates 
that  were  also  expected  to  improve  diagnos¬ 
tics  sometimes  do  not.  Diagnostics  improve¬ 
ments  are  then  scheduled  in  the  next  soft¬ 
ware  or  hardware  update.  These  situations 
often  result  in  limited  diagnostics  data,  in 
an  unprogrammed  extension  to  the  opera¬ 
tional  test  period  to  acquire  the  required 
data,  or  an  inability  to  properly  evaluate  the 
system’s  diagnostics  capability. 

(4)  While  conducting  the  evaluation  is 
the  most  difficult  phase  of  the  test  cycle, 
reporting  the  results  of  the  evaluation  is  the 
more  critical.  The  report  needs  to  convey 
clearly  how  the  diagnostics  performed  and 
what  needs  to  be  done  to  improve  perform¬ 
ance.  The  intent  of  the  report  is  to  suffi¬ 
ciently  inform  decision  makers  so  they  can 
make  an  appropriate  production  decision/ 
direct  funds  and  action  to  fix  identified 
diagnostics  problems.  If  the  report  does  not 
do  this,  the  entire  evaluation  may  have  been 
a  waste  of  time. 

(5)  This  final  section  combines  the  two 
subjects  of  conducting  and  reporting  the  eval¬ 
uation  to  emphasize  the  close  relationship 
they  have  to  each  other.  Decisions  as  to 
what  to  report  and  how  to  structure  the 


13-12 


AFOTECP  400-1  15  May  1991 


report  should  be  made  continuously  through¬ 
out  the  evaluation.  Do  not  wait  until  all  the 
data  are  in  before  beginning  the  report, 
b.  Gathering  and  Analyzing  the  Data: 

(1)  The  first  step  in  conducting  the 
evaluation  is  gathering  data  from  the  opera¬ 
tors  and  maintenance  technicians  who  use 
the  diagnostics  during  the  operational  test 
(see  AFOTEC/LG  operating  instruction  66-1). 
Analyzing  the  data  should  be  a  continuing 
process  that  begins  as  soon  as  data  collection 
starts. 

(2)  Figure  13-3  recaps  the  six  measures 
presented  earlier  and  lists  the  factors  needed 
to  compute  each  measure. 

(3)  The  data  gathering  process  should  be 
structured  and  controlled  to  provide  the 
factors  needed  to  compute  the  diagnostics 
measures.  Unfortunately,  the  standard 
maintenance  data  collection  systems  avail¬ 
able  to  operational  test  teams  will  not  pro¬ 
vide  (in  their  present  form)  all  of  these 
factors.  These  data  collection  systems  are 
the  maintenance  data  collection  (MDC)  sys¬ 
tem  and  the  system  effectiveness  data  sys¬ 
tem  (SEDS).  At  present,  these  two  data 
systems  should  be  supplemented  by  manual 
data  collection  or  by  a  computerized  data 
base  management  system  created  by  or  for 
test  teams  with  access  to  and  control  over 
their  own  data  base. 

(4)  Regardless  of  the  data  system  used, 
data  must  be  generated  from  each  operating 
period  for  each  troubleshooting  maintenance 
action.  The  data  must  be  structured  to 
answer  questions  such  as: 

(a)  What  percentage  of  critical  faults 
were  identified  to  the  operator?  There  are 
only  two  ways  this  can  happen:  manually 
or  automatically. 

(b)  How  were  faults  indicated  to  the 


operator?  By  what  means  are  we  asking 
here? 

(c)  How  many  FAs  did  the  operators 
observe? 

(d)  How  many  CNDs  did  maintenance 
report? 

(e)  How  long  did  it  take  to  trouble¬ 
shoot  malfunctions? 

(f)  Did  BIT  contribute  to  the  fault 
isolation  process?  Qualitative  LRU  fault 
isolation  is  not  even  addressed  in  this  docu¬ 
ment  ever  though  we  often  have  user  re¬ 
quirements.  LRU  fault  isolated  when  used 
in  conjunction  with  percent  of  LRU  failures 
is  a  viable  method  of  evaluating  a  systems 
diagnostic  capability. 

(g)  How  many  I-level  or  D-level  RTOKs 
were  reported? 

(5)  When  tabulated,  answers  to  these 
questions  will  generate  the  factors  for  the 
diagnostics  measures.  Answers  to  the  first 
three  questions  on  critical  failures  and  FAs 
must  come  from  operator  debriefing  records 
or  operator  logs.  Determination  of  critical 
failures  will  be  based  on  the  definition  estab¬ 
lished  from  a  system’s  specific  mission  re¬ 
quirements.  Answers  to  the  fourth  question 
must  come  from  both  operator  and  mainte¬ 
nance  logs.  Some  systems  are  designed  to 
record  and  play  back  anomalies  detected 
during  a  mission,  and  whenever  possible, 
these  recordings  should  be  used  to  analyze 
causes  of  FAs,  CNDs,  and  system  failures. 

(6)  Answers  to  the  last  four  questions 
must  come  from  maintenance  technicians 
and  be  documented  in  the  test  team’s  main¬ 
tenance  data  collection  system. 

(7)  Gathering  and  analyzing  diagnostics 
data  have  some  peculiarities  not  found  in 
routine  maintenance  data  collection  and 


Measures 
%  of  CFI 
UF/OU 

MTT 

Qualitative 

Maintainability 


Description 

Percentage  of  critical  faults  to  the  operators  in  a  clear  and  timely  manner 
Unconfirmed  faults  per  operational  unit  (sorties,  hours,  etc.) 

Mean  time  to  troubleshoot  (troubleshooting  action) 

Qualitatively  assess  integrated  diagnostics  support  elements 


%  of  RTOK  Percentage  of  LRU/SRU  that  RTOK 


Figure  13-3.  Integrated  Diagnostics  Measures  and  Factors 
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analysis.  One  peculiarity  occurs  when  the 
diagnostics  cannot  easily  and  unambiguously 
isolate  a  fault  within  an  ambiguity  group. 
The  technicians  may  be  driven  by  operation¬ 
al  time  constraints  and  resort  to  the  removal 
of  any  suspect  units.  If  troubleshooting 
these  units  is  performed  at  a  higher  main¬ 
tenance  facility  with  some  level  of  automated 
checkouts  equipment,  the  O-level  automated 
diagnostics  (BIT)  cannot  be  credited  with 
aiding  this  fault  isolation  segment.  If  the 
system  remains  down  until  the  malfunction 
is  identified  and  corrected,  it  should  be 
charged  against  O-level  MTT,  MRT,  and 
MDT. 

(8)  Data  relating  to  troubleshooting  times 
should  be  reviewed  carefully  and  frequent¬ 
ly — preferably  daily.  The  daily  screenings 
should  check  for  illogical  entries.  For  ex¬ 
ample,  a  maintenance  data  record  reporting 
an  automated  troubleshooting  action  taking 
15  hours  is  probably  wrong.  Purely  auto¬ 
mated  troubleshooting  would  seldom  take 
long,  so  either  the  time  entry  is  wrong  or 
troubleshooting  was  really  performed  by  a 
combination  of  automated  and  manual  tech¬ 
niques.  In  either  case,  the  apparent  data 
discrepancy  must  be  resolved  quickly  while 
the  technicians  who  performed  the  work  still 
remember  what  actually  happened. 

(9)  Periodically,  throughout  the  evalua¬ 
tion,  the  MTTT  measure  should  be  checked 
for  accuracy  by  comparing  the  MTTT  to  MRT 
or  MDT  times  coming  out  of  an  overall 
maintainability  evaluation.  The  MITT  value 
should  be  less  than  the  MRT  or  MDT  value. 

(10)  There  is  a  continuing  requirement  to 
reassess  the  maintainability  diagnostics 
throughout  the  evaluation.  Early  in  the  test 
period,  technicians  may  spend  a  great  deal 
of  time  troubleshooting  before  concluding 
they  cannot  duplicate  the  reported  malfunc¬ 
tion.  As  testing  progresses,  and  the  tech¬ 
nicians’  experience  increases,  they  may  spend 
much  less  time  troubleshooting  before  they 
sign  off  their  maintenance  as  a  "CND."  This 
tendency  would  result  in  a  decreasing  MTTT 
figure  as  the  test  progresses.  This  decrease 
can  be  attributed  to  the  frustration  involved 
in  what  often  results  in  an  unproductive 
investigation. 

(11)  Near  the  end  of  the  evaluation,  the 
MTTT  figures  should  be  analyzed  from  the 
viewpoint  of  what  will  be  said  in  the  evalua¬ 
tion  report.  If  the  overall  MTTT  figure  is 
satisfactory,  there  may  not  be  much  need  to 
present  more  detailed  data  comparing  mean 
times  for  manual,  BIT  aided,  or  CND  trou¬ 
bleshooting.  However,  if  this  situation  oc¬ 
curs,  carefully  consider  the  conditions  under 
which  troubleshooting  time  data  were  gath¬ 


ered.  Contractor  engineers  or  skilled  test 
team  members  using  nonstandard  work¬ 
around  techniques  could  result  in  a  good 
overall  MTTT  figure,  but  these  results  would 
not  be  realistic  in  a  true  operational  environ¬ 
ment. 

(12)  More  typically,  if  the  diagnostics  do 
not  work  well,  the  results  will  be  higher 
than  the  desired  overall  MTTT  figure.  This 
overall  figure  should  be  amplified  in  the  final 
report  by  comparing  the  other  MTTT  figures 
to  the  overall  figure.  This  comparison  will 
show  the  reductions  in  troubleshooting  time 
which  could  be  expected  if  BIT  could  be  used 
more  effectively  during  all  troubleshooting 
actions. 

(13)  A  further  peculiarity  in  diagnostics 
data  analysis  centers  around  determining 
when  a  unit  retests  OK.  When  O-level 
testing  indicates  a  faulty  unit,  it  usually  is 
retested  at  the  I-  or  D-level  to  confirm  and 
locate  the  fault.  If  testing  fails  at  this 
higher  level  to  confirm  the  fault,  the  unit  is 
classified  as  an  RTOK. 

(14)  As  mentioned  earlier,  there  is  a 
tendency  to  defer  diagnostics  data  gathering 
until  the  latest  software  or  hardware  update. 
However,  if  the  diagnostics  are  at  all  usable, 
data  gathering  and  analysis  should  begin  as 
early  as  possible.  The  data  should  be  seg¬ 
mented  for  each  configuration  change  that 
affects  diagnostics  (block  1,  block  2,  etc.). 
Trend  analysis  of  the  measures  for  each 
configuration  will  show  if  diagnostics  per¬ 
formance  is  improving  as  expected  after  each 
update.  This  trend  analysis  can  also  iden¬ 
tify  needed  diagnostics  improvements. 

(15)  Diagnostics  data  should  be  analyzed 
for  trends  in  FAs,  CNDs,  and  RTOKs.  Trend 
analyses  of  these  measures  may  help  identify 
problems,  causes,  and  solutions. 

(16)  Diagnostics  data  should  also  be 
analyzed  to  identify  the  few  units  that  might 
be  driving  any  adverse  trends — the  "bad 
actors."  Simply  removing  these  units  from 
service  may  significantly  improve  diagnostics 
FA,  CND,  and  RTOK  performance. 

(17)  A  point  made  earlier  but  worth 
repeating  is  the  data  analyses  should  begin 
when  data  collection  starts  and  should  con¬ 
tinue  throughout  the  evaluation.  Continual 
data  analysis  hopefully  can  catch  most  of  the 
errors  in  the  data  collection  process  in  time 
to  prevent  reporting  wrong  conclusions 
drawn  from  erroneous  data. 

c.  Reporting  the  Results: 

(1)  Before  reporting  the  results  of  the 
diagnostics  evaluation,  consider  the  results 
in  context  with  the  evaluations  in  other 
effectiveness  and  suitability  areas  that  are 
dependent  on  the  diagnostics.  Ensure  what 
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you  intend  to  say  concerning  diagnostics  is 
consistent  with  what  is  being  said  in  other 
portions  of  the  report.  For  example,  if  the 
maintainability  evaluation  concludes  that 
system  maintainability  is  good,  your  evalua¬ 
tion  should  have  concluded  that  system 
diagnostics  effectively  supported  the  system’s 
fault  isolation  requirements  or  the  inconsis¬ 
tency  should  be  explained  in  detail.  The 
various  portions  of  the  evaluation  that  are 
dependent  on  the  diagnostics  should  be 
logically  consistent  with  the  diagnostics 
evaluation  and  vice  versa.  If  not,  the  test 
team  must  detail  the  reasons  for  the  incon¬ 
sistency  in  a  logical  and  understandable 
manner. 

(2)  Once  you  have  the  diagnostics  evalu¬ 
ation  results  in  their  proper  perspective,  you 
are  ready  to  begin  writing.  The  clearest  way 
to  present  diagnostics  evaluation  results  is  to 
first  describe  how  the  diagnostics  are  in¬ 
tended  to  work  for  operators  and  mainte¬ 
nance  technicians.  Next,  report  how  they 
actually  worked  by  describing  diagnostics 
deficiencies  determined  during  the  evalua¬ 
tion.  Then  discuss  the  operational  conse¬ 
quences  of  these  deficiencies.  This  approach 
will  have  to  be  adapted  to  the  format  pre¬ 
scribed  for  the  report.  On  the  other  hand, 
the  prescribed  format  should  not  prevent  you 
from  presenting  the  diagnostics  results  in 


AFOTECP  400-1  15  May  1991 


this  manner. 

(3)  Do  not  rely  on  the  quantitative  meas¬ 
ures  by  themselves  to  tell  the  story.  Use 
the  measures  to  quantify  the  story  that 
should  be  presented  in  as  nontechnical  a 
language  as  possible  with  a  minimum  of 
abbreviations. 

(4)  Tables  can  help  present  the  data  as 
long  as  the  significance  of  the  data  is  ex¬ 
plained  in  the  text.  As  examples,  figure  13-4 
illustrates  how  some  of  the  quantitative  data 
can  be  presented  in  the  report,  while  figure 
13-5  illustrates  a  way  of  presenting  trouble¬ 
shooting  data. 

(5)  If  evaluation  results  indicate  the 
integrated  diagnostics  perform  poorly,  and 
the  diagnostics  are  a  significant  factor  in  the 
system’s  supportability,  consider  writing  an 
additional,  more  detailed  report  on  the  diag¬ 
nostics.  The  overall  operational  test  and 
evaluation  report  will  provide  the  broad 
perspective  and  impact  of  poorly  performing 
diagnostics  to  high-level  decision  makers. 
The  more  detailed  report  should  provide 
specific  recommendations  to  a  program  for 
fixing  diagnostics  deficiencies.  To  be  suc¬ 
cessful,  both  types  of  reports  should  clearly 
identify  the  expected  benefits  of  improving 
diagnostics  so  decision  makers  can  determine 
whether  to  spend  money  on  diagnostics  or 
for  other  needed  system  improvements. 


SYSTEM  DIAGNOSTICS  RESULTS 


Measures 

Results 

Criteria 

%  of  critical  faults 
identified  to  operator 

XXX% 

YY.Y9o 

%  of  critical  faults 
identified  to  operator 
by  BIT  (Operafional  Subset) 

XXX% 

YY.Y% 

Unconfirmed  faults  per 
operator  unit 

XXX 

YY/Y 

FA  per  operational  unit 
(CND  per  operational  unit 
optional  subsets; 

XX/X 

YY/Y 

MTT  (O-level  trouble¬ 
shooting) 

XXX  hr 

Y.YY  hr 

%  of  RTOK 

xxx% 

YY.Y% 

Figure  13-4.  Example  of  Table  for  Presenting  Diagnostics  Results 
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SYSTEM  TROUBLESHOOTING  RESULTS 


Troubleshooting 

Techniques 


Mean  Time  to 
Troubleshoot  <Hrs) 


%  of  Diagnostics 
_ Actions 


Only  Manual  Effective 


XXX 


YY.Y 


BIT  Contributed 


XXX 


YYY 


CND 


XXX 


YYY 


Figure  13-5.  Example  of  Table  for  Presenting  Troubleshooting  Results 


JAMES  A.  WILLIAMS,  SMSgt,  USAF 
Director  of  Information  Management 


PETER  D.  ROBINSON 
Major  General,  USAF 
Commander 
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NUCLEAR  HARDNESS  MAINTENANCE/HARDNESS  SURVEILLANCE  (HM/HS) 


Al-1.  Introduction.  The  Air  Force  ex¬ 
pends  many  dollars  of  our  scarce  acquisition 
resources  to  ensure  selected  systems  will  be 
usable  in  transattack  or  postattack  nuclear 
environments.  These  selected  systems  have 
a  nuclear  survivability  requirement.  Surviva¬ 
bility  is  defined  as  the  "capability  of  a  sys¬ 
tem  to  withstand  a  man-made  hostile  en¬ 
vironment  and  be  able  to  accomplish  its 
mission.  Survivability  may  be  achieved  by 
avoidance,  hardness,  proliferation,  reconsti¬ 
tution,  or  any  combination  of  the  above  (AFR 
80-38,  Management  of  the  Air  Force  Surviva¬ 
bility  Program)." 

a.  Avoidance,  proliferation,  and  reconstitu¬ 
tion  are  tactics  or  planning  considerations. 
We  can  plan  to  locate  our  assets  where 
exposure  to  nuclear  environments  is  expected 
to  be  minimal  or  nonexistent  (avoidance). 
We  can  plan  to  buy  many  systems  and 
scatter  them  over  large  areas  expecting  a 
percentage  of  them  to  survive  and  operate 
(proliferation).  We  can  also  make  arrange¬ 
ments  for  spares,  manpower,  support  equip¬ 
ment,  etc.,  to  be  readily  available  to  repair 
expected  nuclear  environment-induced  dam¬ 
age  to  our  systems  (reconstitution). 

b.  Hardness  is  a  design  consideration 
which  must  be  built  into  a  system  to  ensure 
its  ability  to  function  in  expected  nuclear  en¬ 
vironments.  Ideally,  hardiness  is  an  initial 
design  consideration  and  is  planned  for  and 
built  into  a  system  from  the  outset.  Hard¬ 
ness  has  been  retrofitted  into  some  of  our 
older  systems  whose  initial  designs  did  not 
consider  nuclear  threats  or  new  threats 
evolved  that  were  not  anticipated  by  the 
initial  designs.  The  need  and  methods  to 
maintain  and  periodically  verify  a  system’s 
built-in  nuclear  hardness  are  the  objective  of 
an  operational  suitability  assessment  and  will 
be  the  focus  of  this  chapter. 

Al-2.  Explanation  of  Terms: 

a.  Hardness  Maintenance.  Actions 
taken  to  ensure  that  as-produced  system 
hardness  is  not  degraded  over  the  life  cycle 
of  the  system  because  of  maintenance  proce¬ 
dures,  modifications,  or  other  factors  (AFR 
80-38).  It  is  the  procedures  applied  to  en¬ 
sure  that  system  operations,  logistics  support, 
maintenance  actions,  or  natural  causes,  e.g., 
corrosion,  do  not  degrade  the  system’s  de¬ 
signed  and  fielded  hardness  below  acceptable 
levels. 

b.  Hardness  Surveillance.  Measures 
taken  to  evaluate  and  assess  the  hardness  of 
a  fielded  system.  It  is  an  ongoing  process 


throughout  a  system  life  cycle  ;AFR  80-38- 
Hardness  surveillance  includes  specialized 
tests  and  inspections  performed  to  ensure 
design  hardness  is  being  adequately  main¬ 
tained  in  fielded  systems. 

c.  HM/HS  Concept.  Developed  in  the 
early  stages  of  an  acquisition  program,  this 
document  details  the  management  approach 
for  a  weapons  system’s  life  cycle  HM/HS 
program.  It  addresses  the  hardness  design 
approach,  user’s  HM/HS  needs  and  con¬ 
straints,  and  provides  and  overview  of  acqui¬ 
sition  and  operational  responsibilities  for 
HM/HS  requirements.  It  should  address  each 
level  of  maintenance  and  highlight  new  skills 
that  will  be  needed,  HM/HS-peculiar  support 
equipment  that  must  be  developed  or  pro¬ 
cured,  and  related  areas  such  as  training, 
technical  data,  spares,  etc.  Specialized 
system,  subsystem,  LRU,  and  component- 
level  hardness  testing  that  is  planned  or 
required  should  also  be  addressed.  The 
HM/HS  concept  development  is  normally  an 
AFSC  (implementing  command)  responsibilitv. 

d.  HM/HS  Plan.  A  comprehensive  HM 
HS  life  cycle  plan  which  evolves  from  the 
HM/HS  concept.  The  HM/HS  plan  details 
the  HM/HS  program  for  the  production- 
approved  and  fielded  system  and  logistics 
support  infrastructure. 

e.  Hardness  Assurance  Design  Docu¬ 
mentation  (HADD).  A  set  of  technical 
documents  which  provide  detailed  informa¬ 
tion  on  hardness  design,  assumptions,  safety 
margins,  and  specifications.  Developed  by 
the  contractor,  the  HADD  is  maintained  and 
used  primarily  by  the  supporting  command 
to  assess  the  hardness  impacts  of  modifica¬ 
tions,  parts  substitutions,  or  related  actions 
on  fielded  systems. 

f.  Hardness  Critical  Item  (HCI).  A 
hardware  item  at  any  indenture  level  which 
is  critical  to  the  nuclear  hardness  of  the 
system,  subsystem,  or  LRU  in  which  it  is 
employed  and  could  be  degraded  by  improper 
design,  manufacture,  assembly,  modification, 
installation,  removal,  or  repair.  HCIs  typi¬ 
cally  include  items  such  as  EMP  gasketing, 
zener  diodes,  surge  arrestors,  and  other 
specialized  pieceparts  selected  for  their  nu¬ 
clear  hardness  properties. 

g.  Nuclear  Effects.  The  physics  of  a 
nuclear  detonation  and  the  subsequent  effects 
on  the  surrounding  environment  are  complex, 
and  therefore,  a  detailed  explanation  of 
nuclear  effects  of  interest  will  not  be  at¬ 
tempted  here.  Numerous  papers  and  texts 
are  available  to  those  who  are  interested  in 
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the  details  of  nuclear  phenomenology,  and 
one  of  the  most  authoritative,  "The  Effects  of 
Nuclear  Weapons,"  Glasstone,  Samuel  and 
Dolan,  Philip  J.,  1977,  is  available  in  LG3 
for  review.  The  effects  of  primary  interest 
from  a  nuclear  HM/HS  standpoint  are  blast, 
thermal,  transient  radiation  effects  on  elec¬ 
tronics  (TREE),  crew  radiation  dose,  and  elec¬ 
tromagnetic  pulse  (EMP). 

(1)  Blast.  Blast  or  overpressure  is  one 
of  the  commonly  recognized  nuclear  effects. 
It  is  a  strong  pressure  front  caused  by  the 
rapid  expansion  of  hot  gases  from  a  nuclear 
detonation  interacting  with  surrounding 
ambient  air,  water,  or  earth  media.  Blast 
waves  in  air  are  characterized  by  high  veloc¬ 
ity  winds  initially  followed  by  high  negative 
pressures  or  suction.  Blast  wave  velocities 
are  directly  related  to  the  distance  of  the 
observer  from  the  detonation,  height  of  burst, 
and  the  transmission  media;  but  initial  veloc¬ 
ities  may  be  several  times  the  speed  of 
sound.  Blast  effects  can  cause  damages 
ranging  from  total  destruction  to  minor 
distortions,  cracking,  or  bending  of  struc¬ 
tures. 

(2)  Thermal.  Thermal  or  thermal  radia¬ 
tion  is  the  heat  energy  released  from  a 
nuclear  fireball  in  the  form  of  ultraviolet, 
visible,  and  infrared  radiation.  The  incen¬ 
diary  effects  of  a  thermal  pulse  are  the  most 
widely  recognized.  Temperatures  experienced 
can  be  from  several  millions  of  degrees 
Fahrenheit  in  the  immediate  proximity  of  a 
detonation  to  significantly  lesser  amounts 
based  on  the  distance  from  the  explosion  and 
the  absorption  or  scattering  characteristics  of 
the  transmission  medium.  Thermal  effects 
on  systems  can  range  from  total  incineration 
to  structural  weakening  or  thermal  disruption 
of  semiconductor  device  operation. 

(3)  TREE.  Nuclear  radiations  consist  of 
alpha  and  beta  particles,  gamma  rays,  and 
neutrons.  Alpha  and  beta  particles  are  not 
of  great  importance  because  of  their  relative¬ 
ly  short  ranges,  but  gamma  rays  and  neu¬ 
trons  can  pose  significant  threats  to  weapons 
systems  and  living  organisms.  Their  effects 
on  electronics  is  referred  to  as  TREE  and  is 
primarily  observed  as  disruption  or  loss  of 
critical  systems  or  subsystems. 

(4)  Crew  Dose.  This  term  describes  the 
prompt  nuclear  radiation  effect  on  humans 
and  is  expressed  as  the  total  amount  of 
radiation  absorbed  over  a  period  of  time  or 
as  an  exposure  rate,  i.e.,  the  amount  of 
radiation  absorbed  per  unit  time  (usually  per 
hour).  It  is  a  measure  of  prompt  nuclear 
radiation  effects  as  opposed  to  secondary 
effects  such  as  extended  exposure  to  fallout. 
Crew  exposure  can  result  in  minor  discomfort 


to  illness,  loss  of  motor  or  coherent  mental 
functions,  or  death. 

15)  EMP.  Electromagnetic  pulse  effects 
on  weapon  systems  are  responsible  for  the 
majority  of  hardening  design  features  and 
HM/HS  requirements.  EMP  is  basically  a 
high  amplitude,  fast  rise  time,  broadband 
wave  comprised  of  intense  electric  and  mag¬ 
netic  fields  that  radiate  outward  from  a 
nuclear  burst.  The  phenomenon  is  caused 
by  the  collision  of  gamma  rays  and  from  the 
nuclear  burst  with  molecules  in  the  atmos¬ 
phere  which  cause  the  release  of  high  energy 
electrons  (Compton  electrons;.  The  resultant 
wave  is  basically  similar  to  a  radio  wave 
except  in  a  couple  of  major  respects:  The 
EMP  wave  is  significantly  more  intense  than 
a  radio  wave  (EMP  intensities  can  be  on  the 
order  of  50,000  volts  per  meter;,  and  the 
EMP  wave  is  an  extremely  short  pulse 
wherein  virtually  all  its  energy  is  experienced 
in  a  few  nanoseconds  (10  9  seconds).  Actual 
EMP  intensity  is  primarily  a  function  of 
height  of  burst,  weapon  yield,  and  distance. 
Because  EMP  can  be  wide-ranging,  especially 
in  the  case  of  high-altitude  or  ex¬ 
atmospherics  bursts,  Air  Force  systems  on 
the  ground,  in  the  air,  or  in  space  may  be 
affected  by  EMP  without  experiencing  the 
other  effects  described  above.  EMP  effects  on 
Air  Force  systems  can  range  from  temporary 
upset  of  electronic  systems  to  burnout  of 
electronic  devices  and  subsequent  loss  of 
critical  systems  or  subsystems. 

Al-3.  Directives  and  Scope: 

a.  Directives: 

(1)  DODD  4245.4,  Acquisition  of  Nuclear 
Survivable  Systems,  25  July  1988. 

(2)  AFR  80-38,  Management  of  the  Air 
Force  Survivability  Program. 

b.  Scope.  Nuclear  hardness  and  a  life 
cycle  program  to  ensure  hardness  is  a  re¬ 
quirement  for  ail  systems,  major  and  non¬ 
major,  including  modifications,  that  have  a 
nuclear  survivability  requirement.  DODD 
4245.4  specifically  states  that  for  systems 
with  a  nuclear  survivability  requirement. 
"The  acquisition  program  shall  include  devel¬ 
opment  of  a  life  cycle  hardness  program.” 

Al-4.  OT&E  Responsibilities.  Operational 
test  agencies  (OTA)  have  a  responsibility  to 
adequately  assess  a  system  life  cycle  hard¬ 
ness  program  as  they  would  any  other  Air 
Force  and  user  required  capability.  The 
Deputy  Director  of  Defense  for  Operational 
Test  and  Evaluation  (DOT&E)  is  specifically 
tasked  in  DODD  4245.4  to  "...confirm  ade¬ 
quate  assessment  during  OT&E  including 
combined  DT&E/OT&E."  In  addition,  DODD 
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4245.4  requires  the  OT&E  outline  in  test  and 
evaluation  master  plans  (TEMP)  to  "...de¬ 
scribe  how  tests  shall  validate  both  opera¬ 
tional  effectiveness  and  suitability  including 
hardness  maintenance  and  surveillance." 
OTAs  must  ensure  user  life  cycle  hardness 
program  requirements  are  clearly  stated  in 
requirements  documents  such  as  the  mission 
need  statement  (MNS),  operational  require¬ 
ments  document  (ORD),  etc.  Requirements 
must  be  stated  comprehensively  enough  to 
allow  the  implementing  command  to  develop 
a  program  to  meet  the  user’s  needs  and  to 
support  an  adequate  OT&E  assessment. 
After  completing  the  OT&E  assessment  of  the 
system  life  cycle  hardness  program,  the  OTA 
must  include  their  findings  in  the  final 
OT&E  report  for  the  system. 

Al*5.  HM/HS  Assessment  Methodology. 
An  HM/HS  assessment  includes  system 
familiarization  and  documentation  review, 
involvement  in  DT&E  survivability  test 
planning,  final  OT&E  test  plan  and  detailed 
test  procedure  formulation,  test  execution, 
and  reporting. 

a.  Systems  Familiarization  and  Docu¬ 
mentation  Review.  This  is  one  of  the 
cornerstone  tasks  of  HM/HS  assessment 
planning.  It  is  normally  undertaken  by  OTA 
headquarters  personnel  at  the  earliest  stages 
of  a  -stem  acquisition.  The  fundamental 
quest,  ons  that  must  be  comprehensively 
investigated  and  answered  include: 

(1)  Does  the  system  have  a  nuclear 
survivability  requirement?  If  so,  is  the 
requirement  clearly  stated  in  the  MNS  and 
PMD?  Are  the  80  series  AFRs  (specifically 
AFR  80-38)  specified  for  this  acquisition  (see 
the  Authority  and  Deviation  section  of  the 
PMD)? 

(2)  Is  hardness  design  one  of  the  meth¬ 
ods  that  will  be  used  to  ensure  survivability? 
If  so,  is  a  life  cycle  hardness  program  speci¬ 
fied  as  a  user  and  system  requirement?  Is 
the  specified  program  consistent  with  the 
using  command’3  maintenance  concept? 

(3)  Is  a  system  HM/HS  concept  and  plan 
specified  on  contract?  Are  subcontractors 
also  required  to  support  concept/plan  writing 
and  revisions  based  on  the  equipment  they 
provide  and  testing  they  conduct  (this  re¬ 
quirement  is  sometimes  overlooked  and 
almost  always  results  in  the  need  for  a 
contract  modification/renegotiation  as  sub¬ 
contractor  participation  and  data  are  essen¬ 
tial  for  a  complete  and  usable  document)?  Is 
the  OTA  on  distribution  for  the  concept/plan 
deliverables? 

(4)  Is  the  HADD  on  contract?  Are  sub¬ 
contractors  required  to  input  and  revise 


HADD  information  for  their  subsystems.’ 
OTAs  generally  should  not  request  to  be  on 
distribution  for  HADD  deliverables  as  they 
are  usually  voluminous,  of  limited  use  in  the 
planning  and  conduct  of  an  operational  test, 
and  specific  volumes  or  information  car. 
usually  be  readily  accessed  through  the  SPO 
if  required. 

(5)  Does  the  HM/HS  concept  adequately 
scope  the  life  cycle  hardness  program  for  all 
levels  of  maintenance? 

(a)  Does  it  address  user  and  support¬ 
ing  command  requirements/constraints,  and 
is  it  consistent  with  the  system  hardness 
design  (if  the  design  uses  shielded  cables,  are 
provisions  made  for  inspecting/testing  the 
cable  shield;  if  terminal  protection  devices 
such  as  zener  diodes,  surge  suppressors,  etc., 
are  used,  how  will  they  be  tested;  etc.  ? 

(b)  What  will  be  done  at  each  level  of 
maintenance  and  who  will  do  it?  Is  special¬ 
ized  test  equipment  envisioned?  Is  it  availa¬ 
ble  off-the-shelf  or  must  it  be  developed? 
Who  is  responsible  for  procuring  this  equip¬ 
ment  and  related  technical  data,  training, 
spares,  etc.? 

(c)  What  specialized  HM/HS  training 
is  envisioned?  Does  it  include  general  hard¬ 
ness  awareness  training  on  a  recurring  basis 
as  well  as  any  required  technical  training? 
Is  hardness  awareness  training  to  be  pro¬ 
vided  for  operators  and  management  person¬ 
nel  as  well  as  maintainers?  (Hardness 
design  features  and  their  maintenance  are 
generally  transparent  to  normal  peacetime 
operations,  i.e.,  peacetime  operations  will 
normally  be  unaffected  by  degradation/mal¬ 
function  of  hardness  critical  items.  Only- 
through  initial  and  recurring  awareness 
training  of  the  importance  of  these  items  and 
their  maintenance  can  the  survivability  of 
the  system  in  a  nuclear  environment  be 
ensured.) 

(d)  Have  reprovisioning  and  substitu¬ 
tion  controls  for  hardness  critical  items  been 
discussed?  Are  controls  planned  to  ensure 
proposed  substitutions,  configuration  changes, 
or  modifications  are  reviewed  and  approved 
by  a  survivability  engineer  before  implemen¬ 
tation? 

(e)  Have  provisions  been  made  to  collect 
and  analyze  HM/HS  data  from  the  field. 
Will  work  unit  codes  (TO  -06  series;  be 
allocated  so  field  HM/HS  data  can  be  unique¬ 
ly  coded  and  extracted  from  the  maintenance 
data  system  for  analysis?  Who  will  be 
responsible  for  this  analysis?  Will  it  provide 
the  using  and  supporting  command  mainte¬ 
nance  planners  the  information  they  need? 
Will  a  computer  model  be  used  for  individual 
system  or  fleet  probability  of  hardness  predic- 
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tions  based  on  field  HM/HS  and  other  data? 
Who  will  develop  and  maintain  the  model? 
Will  the  model  output  provide  strategic 
planners  the  information  they  need  for  war¬ 
planning? 

(6)  Does  the  HM/HS  concept  and/or  plan 
contain  state  hardness -related  testing  that 
will  be  accomplished? 

(a)  Will  testing  be  done  to  provide 
detailed  visual  inspection  acceptanca/rejection 
criteria  for  HCIs  (this  testing  is  generally 
critical  to  provide  maintenance  personnel 
with  viable  criteria  for  visual  inspection  of 
hardness  elements  such  as  gasketing, 
shielded  cable  overbraid,  etc.)?  Technical 
orders  should  definitively  state  exactly  what 
constitutes  damage  to  these  elements  by 
providing  pictorial  representations  of  ac¬ 
ceptable  and  unacceptable  damage  and/or  by 
providing  measurable  criteria,  e.g.,  any  tear 
in  gasket  may  not  exceed  .5  inch  in  length, 
no  more  than  two  adjacent  fingers  in  finger- 
stock  gaskets  may  be  missing,  no  more  than 
five  fingers  in  tha  total  length  of  fingerstock 
may  be  missing,  etc. 

(b)  Will  testing  be  done  to  establish  a 
threat- relatable  HM/HS  baseline  (hardness 
cit.sign  criteria  and  hardening  features  are 
developed  based  on  nuclear  threat-level 
values,  i.e.,  threat  levels  that  would  be 
anticipated  in  an  actual  nuclear  environ¬ 
ment)?  HM/HS  testing  is  done  at  levels 
typically  much  less  than  anticipated  threat 
levels.  Because  the  low-level  HM/HS  test 
measurements  are  not  directly  relatable  to 
threat-level  measurements,  establishment  of 
a  threat-relatable  baseline  is  generally  de¬ 
sirable.  This  is  typically  done  in  conjunc¬ 
tion  with  threat-level  testirg  at  simulators 
such  as  Trestle,  Horizontally  Polarized  Di¬ 
pole,  or  Vertically  Polarized  Dipole.  Selected 
test  points  on  the  test  article  are  measured 
initially  on  the  threat  simulator  and  then  on 
the  low-level  HM/HS  simulator — frequently  a 
system-level  continuous  wave  (CW)  illumina¬ 
tor.  The  values  from  each  set  of  measure¬ 
ments  can  then  be  compared  and  a  baseline 
for  future  HM/HS  measurements  can  be 
established). 

(c)  Will  testing  be  done  to  verity  the 
utility  of  proposed  commercial  off-the-shelf  or 
developmental  HM/HS  support  equipment? 

b.  Involvement  in  DT&E  Test  Plan¬ 
ning.  Nuclear  effects  testing  is  normally  the 
responsibility  of  development  testers.  OTAs 
provide  support  to  these  test  efforts  to  ensure 
test  articles  are  configured  and  operated  in 
an  operationally  realistic  manner  and  to 
ensure  that  OT&E  data  needs  relating  to  the 
nuclear  effects  testing  are  met.  This  requires 
close  coordination  with  the  DT&E  test  plan¬ 


ners,  typically  in  a  planning  forum  such  as 
the  test  planning  working  group  i.TPWG)  or 
a  survivability/vulnerabiiity  working  group 
(SVWG).  OTAs  may  require  independent 
OT&E  contractor  support  to  assist  them  in 
this  area  of  HM/HS  assessment  because  of 
the  highly  technical  nature  of  the  planning 
and  actual  testing  which  frequently  requires 
an  understanding  of  the  phenomenology', 
simulation/instrumentation  constraints  and 
limitations,  and  test  data  analyses.  This 
contract  support  is  often  obtained  through  a 
joint  effort  with  the  Operations  Analysis  tOA. 
Directorate  who  also  frequently  rely  on 
contract  support  for  investigating  effective¬ 
ness  issues  relating  to  the  nuclear  effects 
testing.  LG3  has  a  memorandum  :f  agree¬ 
ment  on  file  which  delineates  the  responsibil¬ 
ities  of  LG  and  OA  personnel  in  obtaining 
and  managing  contract  support  for  survivabil¬ 
ity  test  planning.  One  of  the  deliverables 
often  required  of  the  OT&E  contractor, 
primarily  on  major  weapon  systems,  is  a 
nuclear  assessment  plan  (NAP).  The  NAP  is 
a  structured  approach  to  OT&E  nuclear 
assessment  planning  that  is  generally  con¬ 
ducted  in  two  or  more  phases.  AFOTEC/OA 
Technical  Paper  13.0,  April  1988,  should  be 
addressed  in  the  DT&E  nuclear  effects  test 
planning  phase.  HM/HS  assessment  factors 
to  be  addressed  include: 

(1)  Is  a  nuclear  assessment  plan  to  be 
developed?  If  so,  the  logistics  evaluation 
manager  must  work  closely  with  his  OA 
counterpart  to  ensure  all  HM/HS  and  effec¬ 
tiveness  issues  are  addressed  by  the  NAP 
and  the  NAP  is  structured  to  support  rele¬ 
vant  critical  operational  issues  and  OT&E 
test  objectives. 

(2)  Will  an  independent  contractor  be 
used  by  the  OTA  to  assist  in  DT&E  test 
planning  reviews,  assessments,  and  recom¬ 
mendations.'  If  so,  the  logistics  evaluation 
manager  must  provide  his  OA  counterpart 
with  HM/HS  assessment  requirements  to  be 
included  in  contract  subtask  statements. 
Contractor  assistance  is  normally  valuable  in 
the  following  areas: 

(a)  Comparison  of  user  HM/HS  re¬ 
quirements  and  maintenance  concepts,  HM/ 
HS  concepts,  and  related  information  with 
the  overall  DT&E  nuclear  effects  test  plan¬ 
ning  program  to  identify  shortfalls,  inconsist¬ 
encies,  or  omissions. 

(b)  Comparison  of  planned  DT&E  test 
points  with  mission  critical  or  suspected 
nuclear  effects- sensitive  systems,  subsystems, 
or  LRUs.  A  limited  number  of  test  points 
can  be  measured  in  any  given  test  effort. 
Care  must,  therefore,  be  exercised  to  ensure 
the  most  critical  test  points  are  given  priority 
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in  the  test  planning  effort. 

(c)  Review  of  DT&E  test  planning  to 
ensure  planning  includes  visual  inspection 
acceptance/rejection  criteria  testing  and 
establishment  of  quantifiable,  threat-relatable 
HM/HS  baseline  (see  paragraphs  5a(6)(a)  and 
(b)  above).  If  the  system  to  be  tested  is 
already  in  OT&E,  valuable  information 
relating  to  which  HCIs  are  failing  frequently 
because  of  inherent  or  induced  causes  can  be 
obtained  by  interviewing  test  team  personnel. 
These  HCIs  should  be  given  special  attention 
by  the  DT&E  testers  as  they  will  probably  be 
high  failure  items  in  the  fielded  system  and 
will  require  frequent  inspections. 

(d)  Thorough  review  and  analysis  of 
planned  HM/HS  baseline  testing.  The  base¬ 
line  testing  must  provide  a  quantifiable 
"measuring  stick"  that  can  be  used  on  fielded 
systems.  Numbers,  types,  and  locations  of 
baseline  test  points  should  be  reviewed  to 
ensure  they  are  reasonable  based  on  test 
equipment  to  be  used  in  the  field,  technicians 
that  will  be  performing  the  tests,  time  that 
will  be  allowed  for  testing,  expected  reliabil¬ 
ity  of  the  area  being  tested  (are  we  measur¬ 
ing  the  areas  where  we  expect  problems),  etc. 
In  addition,  the  technical  soundness  of  the 
approach  used  to  relate  the  low-level  meas¬ 
urements  to  threat  level  and  the  establish¬ 
ment  of  comprehensive  low-level  pass-fail 
criteria  should  be  assessed. 

(3)  Is  the  planned  DT&E  testing  as 
operationally  realistic  as  possible?  Are 
support  equipment,  GFE,  pylon-mounted 
weapons,  deployed  antennas,  etc.,  that  will 
be  used  or  connected  to  the  system  in  an 
operational  environment  planned  for  use  in 
the  DT&E  tests?  Some  items  that  may  be 
connected  in  a  normal  operating  mode,  e.g., 
a  power  cart  on  an  alert  aircraft,  can  pose  a 
significant  energy  penetration  into  an  other¬ 
wise  hardened  system.  In  addition,  some 
items  of  support  equipment  that  may  be 
critical  to  the  operation  of  the  system  under 
test  may  not  have  initially  been  developed  as 
hardened  items.  To  emulate  the  operational 
environment  as  much  as  possible  and  to  test 
the  potential  "worst-case"  scenarios,  careful 
attention  must  be  given  to  realistic  operation¬ 
al  configurations.  Regular  participation  in 
TPWGs  and/or  SVWGs  is  necessary  to  ensure 
proposed  test  configurations  are  realistically 
planned  and  required  assets  are  programmed 
for  well  in  advance. 

c.  OT&E  Plan  and  DMAP.  HM/HS  is 
normally  addressed  as  a  stand-alone  objec¬ 
tive,  an  MOE  of  a  nuclear  survivability 
objective  (effectiveness  issues  constitute  other 
MOEs),  or  it  is  addressed  under  logistic 
support  objectives.  An  example  of  an  HM/HS 


dedicated  test  objective  is  'assess  the  ade¬ 
quacy  of  the  System  X  nuclear  hardness 
program  to  ensure  system  hardness  through¬ 
out  the  System  X  life  cycle.  ’  Regardless  of 
how  the  test  plan  is  structured,  the  basic 
areas  of  assessment  are  usually  the  same. 
Major  areas  of  interest  and  questions  that 
should  be  considered  in  test  plan  and  DMAP 
formulation  include: 

(1)  Technical  Data.  Are  HCIs  and  HCPs 
properly  marked  in  the  TOs  (usually  in 
accordance  with  MIL-STD  lOOC/Engineering 
Drawing  Practices)?  Are  the  proper  tools 
and  support  equipment  cited  in  the  TOs  for 
HCPs.  Are  definitive  pass-fail  criteria  pro¬ 
vided  for  visual  inspections  and  for  manual, 
semiautomated,  or  automated  surveillance 
testing?  Have  provisions  been  made  for 
unique  coding  of  hardness-related  main¬ 
tenance  actions  and  failure  data? 

(2)  Training.  Have  provisions  been  made 
for  initial  and  recurring  hardness  awareness 
training  for  operations,  maintenance,  manage¬ 
ment,  and  support  (supply,  procurement,  etc., 
personnel?  Is  this  training  consistent  with 
system  hardness  design  and  the  life  cycle 
hardness  management  program  developed  for 
the  system?  Will  the  training  program  be 
fully  developed  and  in  place  to  support 
fielding  of  the  system? 

(3)  Supply  Support.  Have  procedures 
been  developed  to  ensure  HCIs  are  not 
substituted  for  at  base  or  depot  level  without 
the  approval  of  the  cognizant  survivability- 
engineering  office?  Has  a  HADD  or  similar 
document  been  developed  to  document  HCI 
specifications,  margins,  and  assumptions  to 
support  survivability  engineering  in  parts 
selection? 

(4)  Support  Equipment  (SE).  Is  required 
HM/HS-peculiar  SE  available?  If  not,  has 
responsibility  for  procurement  of  HM/HS- 
peculiar  SE  been  assigned  and  agreed  to? 
Has  the  SE  been  tested  and  verified  as 
adequate?  Can  it  be  used  in  the  environ¬ 
ments  and  by  the  personnel  specified  in  the 
maintenance  concept?  Have  necessary  tech¬ 
nical  data  been  developed  for  the  SE?  Have 
definitive  pass-fail  criteria  been  provided? 
Does  the  SE  meet  portability  or  transporta¬ 
bility  requirements,  as  applicable? 

(5)  Other  Areas: 

(a)  HM/HS  Data  Management.  Has  a 
using  or  supporting  command  office  been 
assigned  responsibility  to  extract,  analyze, 
and  report  on  field  HM/HS  data?  Have 
provisions  been  made  to  implement  revisions, 
modifications,  or  other  changes  to  inspection 
intervals  or  types  of  inspections  based  on 
HM/HS  data  analyses? 

(b)  Hardness  Configuration  Control. 
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Have  provisions  been  made  to  ensure  any 
proposed  modifications  or  changes  to  the 
system  are  reviewed  and  approved  by  the 
cognizant  survivability  engineering  office 
before  implementation? 

(c)  Hardness  Computer  Model.  Have 
provisions  been  made  for  development  of  a 
system  model,  if  applicable,  to  formulate 
individual  asset  or  fleet  hardness  estimates 
based  on  field  HM/HS  data?  Will  the  model 
be  available  to  support  system  IOC?  Are 
model  outputs  adequate  and  appropriate  for 
anticipated  users  of  the  data? 

d.  Test  Execution  and  Reporting. 
HM/HS  test  execution  and  reporting  are 
normally  joint  headquarters  and  test  team 
efforts.  Early  HM/HS  program  and  require¬ 
ments  documentation  reviews  and  early 
DT&E  test  planning  involvement  are  usually 
done  by  headquarters  personnel.  As  the  test 
team  is  assigned  and  becomes  familiar  with 
HM/HS  objectives,  they  assume  more  of  the 
assessment  responsibilities  and  are  normally 
responsible  for  final  assessment  in  all  the 
HM/HS  areas.  Final  reporting  is  accom¬ 
plished  as  structured  in  the  test  plan,  either 
addressed  as  results  to  a  specific  objective  or 
as  results  to  specified  MOEs. 


Al-6.  HM/HS  Office  of  Primary  Respon¬ 
sibility  and  HM/HS  References.  The 
Logistics  Directorate  OPR  for  HMHS  is  LG3. 
References  maintained  in  LG3  that  are  useful 
for  OT&E  HM/HS  assessment  planning  and 
nuclear  effects  familiarization  include  the 
following: 

a.  The  Effects  of  Nuclear  Weapons,  Glass- 
tone  and  Dolan,  1977. 

b.  B-1B  EMP  Nuclear  Assessment  Plan. 
Booz-Allen  &  Hamilton,  Document  No.  09006- 
141/102-02/0026,  30  September  1986. 

c.  B-1B  Hardness  Maintenance/Hardness 
Surveillance  Plan,  Rockwell  International. 
NA-86-1587,  20  November  1986. 

d.  Nuclear  Hardness  and  Survivability 
OT&E  Methodologies  for  Air  Force  Tactical 
Systems,  The  BDM  Corporation,  BDM/ABQ- 
86-0889-TR,  October  1986. 

e.  Videotape,  "Hardness  Maintenance:  It 
Depends  On  You,"  OL-ALC. 

f.  Videotape,  "Nuclear  Survivability: 
Introduction  to  Hardness  Maintenance,"  OC- 
ALC. 

g.  Videotape,  "Assessing  the  Nuclear 
Survivability  of  Aircraft,"  OC-ALC. 

h.  LG  Training  Course,  "Nuclear  HMHS 
Assessment,"  LG3. 
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DORMANT  RELIABILITY 


A2-1.  Introduction: 

a.  Purpose.  This  chapter  summarizes 
the  current  state  of  the  art  in  identifying  the 
effects  of  dormancy  as  it  relates  to  opera¬ 
tional  reliability.  It  does  not  provide  an 
exhaustive  treatment  of  dormant  reliability. 
The  chapter  provides  a  framework  in  which 
to  develop  an  approach  to  evaluate  the 
effects  of  dormancy  on  a  specific  system. 

b.  Background: 

(1)  The  reliability  of  military  systems 
after  long  periods  of  dormancy  has  been  a 
major  concern  throughout  military  history. 
A  system  taken  out  of  storage  is  expected  to 
accomplish  its  mission  within  some  accep¬ 
table  degree  of  performance  degradation.  In 
early  military  history,  spoilage  of  items  such 
as  food  and  gunpowder  was  a  major  concern. 
A  few  years  ago,  when  aircraft  availability 
exceeded  flying  requirements,  care  was  taken 
to  periodically  move  parked  aircraft  to  miti¬ 
gate  the  effects  of  nonuse  (e.g.,  flat  tires, 
fluid  drain,  etc.).  As  military  systems  contin¬ 
ued  to  become  more  sophisticated,  complex, 
and  expensive,  and  as  their  response  time 
has  become  shorter,  the  need  for  higher 
reliability  has  increased.  Inherent  in  that 
need  is  a  requirement  for  higher  dormant 
reliability. 

(2)  AFOTEC  is  involved  in  the  opera¬ 
tional  test  and  evaluation  (OT&E)  of  a  num¬ 
ber  of  weapon  systems — principally  muni¬ 
tions — that  spend  extensive  periods  of  time 
in  storage.  Historically,  these  systems  have 
exhibited  relatively  high  reliability,  but  on 
newer  systems  complexity  is  increasing, 
longer  service  lives  are  required,  and  periodic 
maintenance  and  checkouts  are  being  reduced 
or  eliminated.  Therefore,  concern  about  the 
effects  of  dormancy  on  a  system’s  operational 
reliability  is  growing,  and  development  of  an 
approach  for  assessing  dormant  reliability  as 
part  of  the  test  and  evaluation  process  is 
becoming  increasingly  important. 

c.  Structure.  The  next  paragraph  pre¬ 
sents  a  compilation  of  pertinent  concerns 
about  dormancy  and  establishes  the  need  for 
considering  its  effects.  Paragraph  A3-3  then 
discusses  the  characteristics  of  the  dormancy 
problem.  Paragraph  A2-4  addresses  some 
considerations  relevant  to  the  assessment  of 
dormant  reliability,  and  paragraph  A2-5 
summarizes  documented  methodologies  for 
estimating  dormant  reliability.  Specific 
assessment  tools  and  techniques  are  dis¬ 
cussed  in  paragraph  A2-6,  an  approach  to 
dormant  reliability  follows  in  paragraph  A2-7, 
and  paragraph  A2-8  concludes  with  a  sum¬ 


mary  of  lessons  learned  from  specific  pro¬ 
grams. 

A2-2.  The  Need  for  Considering  Dor¬ 
mancy  Effects: 

a.  Dormancy  and  Weapon  Systems. 
The  sophistication  and  complexity  of  modern 
weapons  coupled  with  the  rapid  response 
time  required  to  effectively  counter  the 
expected  threat  preclude  extensive  checkout 
and  repair  prior  to  employment.  In  particu¬ 
lar,  dormancy  in  munition  systems  is  a 
concern  because  these  systems  spend  the 
majority  of  their  life  in  a  nonoperating 
environment  (e.g.,  containerized/noncontainer- 
ized  storage,  alert,  etc.).  Also,  many  subas¬ 
semblies  of  munitions  do  not  operate  during 
captive  carry.  The  wooden  round  mainte¬ 
nance  concept  under  which  munitions  are 
accepted  and  deployed  to  operational  units  as 
"all-up  rounds"  with  minimal  field-level 
checkout  and  maintenance  clearly  increases 
the  ratio  of  nonoperating  time.  In  a  typical 
munition  system,  even  with  periodic  check¬ 
out,  nonoperating  time  could  be  as  much  as 
two  million  times  longer  than  operating  time. 
While  such  a  large  ratio  of  nonoperating  to 
operating  time  may  not  be  the  case  for  other 
types  of  weapons,  the  fact  that  most  weapons 
do  spend  considerable  time  in  a  dormant 
state  makes  dormancy  a  major  factor  to 
consider  when  attempting  to  estimate  or 
project  a  mature  system’s  operational  reliabil¬ 
ity. 

b.  Policy,  Guidance,  and  Direction: 

(1)  Only  limited  guidance  exists  for  the 
design  and  conduct  of  testing  for  dormant 
reliability.  Reliability  testing  is  addressed  in 
a  general  sense  under  operational  suitability 
in  OMB  Circular  A- 109,  Major  System  Acqui¬ 
sitions,  the  DOD  5000  series  directives,  and 
AFR  800-18.  MIL-STD  1388,  Logistics  Sup¬ 
port  Analysis,  mentions  dormant  reliability 
by  stating  that  the  logistics  support  analysis 
for  reliability  factors  provides  data  for 
"...effects  of  storage,  shelf  life....”  The  data 
input  to  the  LSA  comes  from  MIL-STD  785, 
Reliability  Program  for  Systems  and  Equip¬ 
ment  Development  and  Production,  reliability 
programs.  MIL-STD  785  discusses  adminis¬ 
trative  requirements  and  general  guidance  for 
reliability  testing  but  provides  no  specific 
guidance  for  dormant  reliability  assessment. 

(2)  Rome  Air  Development  Center  (RADC) 
studies  related  to  nonoperating  failures  have 
consistently  concluded  that  government 
documents  establishing  and  supporting  relia¬ 
bility  requirements  should  be  upgraded  to 
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include  provisions  for  nonoperating  mode 
reliability  requirements  and  predictions. 
Degradation  effects  in  various  dormancy 
states  (e.g.,  operationally  ready  (OR)  storage, 
handling  and  transportation,  launcher  car¬ 
riage,  alert,  captive  carry)  must  be  considered 
in  addition  to  those  of  the  normally  energized 
(active)  state. 

A2-3.  Characterization  of  the  Dormancy 
Problem: 

a.  Dormancy: 

(1)  Dormancy  is  defined  as  those  states 
in  which  a  system  is  not  operating  or  is 
being  maintained  in  OR  storage — including 
all  on-equipment  maintenance  and  functional 
checks  or  built-in  test  (BIT)  necessary  to 
maintain  the  desired  status.  Dormancy  is 
defined  at  the  subsystem  level  when  those 
subsystems  are  installed  on  the  complete 
weapon  system.  This  concept  is  logical  in 
that  some  subsystems  may  be  totally  inactive 
(e.g.,  missile  rocket  motors)  while  other  sub¬ 
systems  (e.g.,  missile  guidance  units)  may  be 
operating  during  various  phases  of  the  sys¬ 
tem’s  life  cycle  prior  to  completing  a  mission. 

(2)  For  tactical  munitions,  the  weapon 
may  rotate  through  various  operational 
postures — OR  storage  to  alert,  OR  storage, 
captive  carry,  etc.  Strategic  systems  may 
spend  long  periods  of  time  in  an  alert  status 
with  guidance  systems  operating  and  the 
remainder  of  the  subsystems  totally  inactive. 

(3)  The  OR  storage  mode  is  predominant 
in  that  it  is  in  this  state  where  reliability 
degradation  caused  by  dormancy  is  most  ap¬ 
parent.  The  ability  of  a  system  to  with¬ 
stand  OR  storage  may  be  influenced  by 
relatively  short  periods  of  operating  time  or 
the  stress  inherent  in  other  states,  such  as 
transportation.  Conversely,  the  ultimate 
operational  reliability  is  influenced  by  the 
ability  to  withstand  long  periods  of  dorman¬ 
cy. 

b.  Dormancy  Effects: 

(1)  Failures  experienced  during  dorman¬ 
cy  have  basically  the  same  effect  as  other 
failures  on  logistics.  Required  spares  provi¬ 
sioning,  manpower  and  inspection  interval 
requirements,  and  their  associated  costs 
throughout  a  systems’s  life  cycle  are  affected 
by  dormant  failures  and  must  be  correctly 
estimated  if  the  system  is  to  be  adequately 
supported.  The  effects  of  dormancy  on 
operations  are  more  closely  related  to  the 
capability  of  the  system  to  function  effective¬ 
ly.  Not  all  failures  are  critical.  Those  which 
are  critical  certainly  affect  operations;  those 
which  are  not  may  not  impact  operations  but 
they  directly  affect  logistics. 

(2)  The  task  for  the  analyst  is  to  deter¬ 


mine  those  failures  which  directly  affect 
operational  capability.  For  example,  suppose 
a  certain  seal  tends  to  dry  out  and  crack 
after  prolonged  periods  of  dormancy.  A 
failure  mode  and  effects  analysis  >FMEA 
would  conclude  that  hydraulic  fluid  leaks 
because  the  seal  cracks.  The  logistics  ana¬ 
lyst  would  conclude  that  more  spare  missiles 
will  be  required  to  support  the  wooden  round 
maintenance  concept  because  the  hydraulic 
actuation  system  leaks  during  storage.  The 
operations  analyst  would  conclude  that  the 
missile  will  probably  miss  the  target  because 
the  hydraulics  system  fails  to  drive  the 
control  surfaces.  The  effects  of  dormancy 
must  be  addressed  at  a  level  which  permits 
estimation  of  their  impact  on  operationai 
effectiveness  and  suitability. 

c.  Inherent  Limitations: 

(1)  There  are  several  limitations  inher¬ 
ent  in  the  nature  of  the  dormancy  problem. 
First,  when  a  system  fails  during  dormancy, 
it  is  extremely  difficult  to  know  when  the 
failure  occurs.  Second,  if  checks  of  the 
system  are  performed  during  dormancy,  those 
checks  may  induce  a  failure.  Third,  it  is 
generally  difficult  to  tell  what  caused  a 
failure-age,  transportation  stresses,  manufac¬ 
turing  defects,  and  induced  maintenance 
failures  could  be  some  of  the  candidates. 

(2)  Measuring  the  effects  of  dormancy 
may  require  calendar  time  in  excess  of  that 
scheduled  for  a  typical  operating  system  in 
test  and  evaluation.  As  such,  dedicated 
dormancy  programs  during  OT&E  are  usually 
not  planned.  Rather,  AFOTEC  piggybacks 
off  AFSC/AFLC-initiated  programs  (e.g., 
warranty  programs,  shelf-like  extension 
programs,  surveillance  programs,  etc.). 

d.  Interactive  and  Long-Term  Nature 
of  the  Process: 

(1)  An  alternative  for  actual  measurement 
of  dormancy  is  to  pursue  the  development  of 
early  estimating  projection  methodologies 
(even  rules  of  thumb)  and  test  methodologies 
that  can  be  improved  over  time.  In  addition, 
AFOTEC  involvement  and  coordination  with 
all  system  development  participants  are 
essential  so  early  program  data,  tests,  and 
surveillance  programs  can  be  structured  in 
a  mutually  supportive  way.  Detailed  knowl¬ 
edge  of  other  methods,  even  though  they 
might  not  be  applicable  to  OT&E,  and  knowl¬ 
edge  of  weapon  systems  in  general  are  both 
necessary  to  improve  the  process.  Also, 
AFLC  may  conduct  special  test  and  evalua¬ 
tion  programs  on  fielded  systems,  the  final 
reports  of  which  are  available  from  the 
system  program  manager  at  the  managing 
ALC. 
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A2-4.  Dormant  Reliability  Assessment 
Considerations: 

a.  Dormant  Reliability: 

(1)  Paragraph  A2-3  discussed  the  charac¬ 
teristics  and  effects  of  dormancy  on  weapon 
systems.  Now,  the  focus  is  on  measurement 
of  those  effects  and  assessment  of  their 
impact  on  system  reliability. 

(2)  Reliability  is  the  probability  that  an 
item  will  remain  failure  free  under  specified 
conditions  over  a  specified  period  of  time  or 
be  in  a  failure-free  state  after  a  specified 
period  of  time.  In  considering  dormant 
reliability,  the  same  reliability  definition 
applies  except  that  "over  time"  covers  several 
possible  states  which  exhibit  potentially 
different  failure  characteristics.  Thus,  a  full 
treatment  of  dormant  reliability  must  con¬ 
sider  all  possible  states — inherent  dormant 
reliability,  OR  storage  reliability,  and  the 
reliability  associated  with  other  periods  of 
time  when  the  system  is  not  operating. 

(3)  Therefore,  dormant  reliability  is 
defined  as  the  probability  that  an  item  will 
remain  failure  free  for  a  specified  period  of 
time  in  a  nonoperating  mode  under  stated 
environmental  conditions.  When  the  system 
is  removed  from  the  dormant  stage,  it  is 
expected  to  perform  within  specifications. 

b.  Acquisition  Process  Considerations: 

(1)  The  AFOTEC  planning  process  will 
usually  begin  near  Milestone  0  and  continue 
beyond  Milestone  II  in  the  acquisitions 
process.  Early  in  the  conceptual  phase, 
reliability  specifications  will  be  formulated  by 
the  developing  command  from  the  using  com¬ 
mand’s  operational  requirements.  Depending 
on  the  input  to  the  dormant  reliability  as¬ 
sessment  approach,  some  initial  insight  into 
dormant  reliability  requirements  could  be 
obtained  during  the  conceptual  phase.  Data 
for  any  assessment  will  come  primarily  from 
the  system  program  office  (SPO)  and  the 
developing  contractors  and  will  generally  be 
limited  to  specifications  and  preliminary 
designs. 

(2)  As  the  program  progresses  into  the 
validation  phase,  data  should  be  available 
from  failure  modes  analyses,  design  reviews, 
initial  reliability  evaluation  tests,  and  failure 
analyses.  Again,  the  SPO  and  contractor  will 
be  the  sources  of  the  information.  From  the 
operational  tester’s  viewpoint,  the  usability 
of  the  data  will  depend  on  understanding 
how  it  was  obtained.  For  tactical  missiles, 
specialized  data  systems  at  WR-ALC  and 
OO-ALC  are  also  a  source  of  historical  infor¬ 
mation. 

(3)  During  IOT&E,  operational  failure 
data  will  be  available,  and  some  dormant 
failure  data  may  be  available  from  surveil¬ 


lance  program,  assuming  that  such  programs 
were  initiated  early.  Attention  must  be  paid 
to  the  nature  and  characteristics  of  data 
gathered  during  various  phases  of  OT&E  to 
ensure  that  any  differences — based,  for  ex¬ 
ample,  on  preproduction  versus  production 
versions  of  the  system — are  understood  and 
accounted  for. 

c.  System-Specific  Considerations. 
System-specific  considerations  are  directly 
related  to  the  system  hardware,  its  intended 
operating  environment,  the  operational  and 
maintenance  concepts,  and  the  system's 
projected  life  cycle  profile.  A  thorough  un¬ 
derstanding  of  each  of  these  areas  is  required 
to  effectively  assess  the  effects  of  dormancy 
and  project  dormant  reliability  for  the  given 
system. 

d.  Test  and  Evaluation  Considera¬ 
tions.  Some  considerations  include  inherent 
limitations  in  the  OT&E  process,  the  role  of 
sampling  and  its  relationship  to  OT&E. 
methodologies  for  projecting  mature  system 
dormant  reliability  based  on  OT&E  results, 
and  capabilities  for  verifying  or  establishing 
the  validity  of  early  reliability  predictions. 
A  brief  discussion  of  each  of  these  follows: 

(1)  Inherent  Test  Limitations.  Major 
limitations  during  OT&E  are  usually  resource 
related.  That  is,  when  the  amount  of  time 
and  assets  devoted  to  test  are  limited,  a 
thorough  and  adequate  assessment  of  dor¬ 
mant  reliability  is  extremely  difficult  to 
accomplish.  Potential  dormant  reliability 
assessment  problems  can  be  reduced  through 
early  involvement  and  careful  planning. 

(2)  Sampling: 

(a)  Because  it  is  not  feasible  to  obtain 
failure  rate  measurements  on  entire  popula¬ 
tions  of  weapon  systems,  the  techniques  of 
reliability  assessment  rest  on  statistical 
concepts.  Such  techniques  permit  the  extrap¬ 
olation  of  results  obtained  from  a  sample  to 
the  total  population  and  possibly  to  other 
similar  populations.  Selection  of  an  appropri¬ 
ate  sample  of  systems  for  testing  depends  on 
the  hypothesis  to  be  addressed  and  the 
potential  risks  associated  with  accepting  or 
rejecting  the  test  results. 

(b)  Determination  of  the  test  sample 
size  also  depends  on  the  planned  test  meth¬ 
od.  Two  commonly  used  methods  for  dor¬ 
mant  reliability  testing  are  fixed-length  tests 
and  tests  terminated  after  a  specified  number 
of  failures. 

(3)  Projection  Methodologies: 

(a)  The  issue  of  dormant  reliability  is 
part  of  a  substantially  larger  problem — pro¬ 
jecting  mature  system  reliability  early  in  the 
life  cycle.  There  are  two  aspects  to  the 
projection  problem,  although  the  difference, 
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while  real,  may  be  very  subtle.  There  must 
be  an  initial  estimate  of  system  reliability 
while  later  projections  tend  to  be  refinements 
of  earlier  projections. 

(b)  During  the  planning  phase,  meth¬ 
odologies  will  generally  be  restricted  to  those 
which  can  use  nonmeasurement  or  limited 
part  test  data.  Contractor  predictions,  judg¬ 
ment,  comparability  analyses,  and  simula¬ 
tion/modeling  are  the  primary  tools.  During 
IOT&E,  as  test  results  become  available, 
regression  and  surveillance  and  inspection 
methods  can  be  added.  These  methodologies, 
with  the  exception  of  contractor  predictions, 
will  still  be  applicable  during  FOT&E. 

(4)  Validation  and  Verification  (V&V). 
V&V  of  dormant  reliability  assessments  is  a 
difficult  and  time-consuming  task.  Field 
results  will  quite  often  require  5  to  10  years 
of  exhaustive  measurement,  data  collection, 
and  analysis  before  dormant  reliability  pre¬ 
dictions  can  be  verified.  Since  the  verifica¬ 
tion  process  provides  the  empirical  feedback 
necessary  to  help  validate  the  reliability 
projection  methodology,  the  validation  process 
is  also  accomplished  over  an  extended  period 
of  time.  Experience  and  data  from  similar 
systems  can  be  used  in  the  V&V  process,  but 
care  must  be  exercised  to  ensure  applicability 
of  those  data. 

A2-5.  Assessment  Methodologies: 

a.  Current  Methodologies.  Current 
techniques  for  estimating  dormant  reliability 
generally  fall  under  the  three  rather  broad 
approaches  of  analytical  prediction  based  on 
parts  count  and  stress  analysis,  failure  rate 
modification  factors,  and  testing.  Each  of 
these  approaches  has  advantages  and  disad¬ 
vantages  which  warrant  the  cautious  use  of 
their  results.  A  brief  discussion  of  the  three 
approaches  follows: 

(1)  Parts  Count  and  Stress  Analysis 
Prediction.  The  parts  count  reliability  predic¬ 
tion  method  assumes  that  the  equipment 
failure  rate  is  a  function  of  the  failure  rates 
of  its  components  or  parts.  This  method  is 
most  applicable  during  the  early  design 
phase  since  it  permits  relatively  easy  compar¬ 
ison  and  evaluation  of  alternative  designs. 
It  is  not  likely  that  the  operational  tester 
will  use  this  prediction  method  to  estimate 
dormant  reliability,  but  the  system  being 
developed  will  probably  base  early  reliability 
predictions  on  this  technique. 

(2)  Failure  Rate  Modification  Factors 
Prediction: 

(a)  Numbers  which  are  used  to  modify 
failure  rates  to  account  for  varying  stresses 
imposed  by  different  applications  and  envi¬ 
ronments  are  generally  known  as  failure- 


rate  modification  or  K"  factors.  They  arc- 
used  to  adjust  a  basic  'known  '  failure  rate 
for  hardware  when  directly  applicable  experi¬ 
ence  data  are  not  available.  Factors  have 
been  developed  and  applied  at  all  levels  from 
generic  parts  to  total  systems. 

(b)  Failure  rate  predictions  derived 
from  K  factors  should  be  less  accurate  than 
rates  derived  from  directly  comparable  expe¬ 
rience.  K  factors  probably  have  more  appli¬ 
cability  at  the  part  or  component  level.  At 
best,  K  factors  may  provide  ballpark  relation¬ 
ships  between  application  environments. 

(3)  Testing: 

(a)  Testing  provides  the  mechanism  for 
obtaining  empirical  failure  rate  data.  There 
are  two  driving  considerations  (discussed 
earlier)  in  any  test  program:  sample  size 
and  required  time.  Two  primary  methods  of 
testing  are  accelerated  testing,  which  pro¬ 
vides  failure  rate  data  in  a  relatively  short 
period  of  time,  and  surveillance  testing, 
which  is  done  in  real  time  and  is  more 
representative  of  the  operational  environ¬ 
ment. 

(b)  Accelerated  testing  is  of  minimal 
use  in  dormant  reliability  because  failures 
associated  with  dormancy  may  not  occur 
during  some  short-time  interval.  This  type 
of  testing  also  has  not  been  successful  at  the 
system  level  because  of  difficulties  associated 
with  constructing  appropriate  acceleration 
factors. 

(c)  Surveillance  testing  generally  re¬ 
quires  that  a  number  of  preproduction  or 
production  systems  be  placed  in  actual  or 
simulated  field  storage  conditions.  Periodical¬ 
ly,  selected  samples  of  these  assets  are 
removed  from  storage  and  examined  for 
degradation  from  original  specifications.  The 
surveillance  program’s  value  to  the  operation¬ 
al  tester  lies  in  the  availability  of  similar 
system  data  upon  which  to  base  a  compara¬ 
bility  analysis  when  developing  an  early  sys¬ 
tem  reliability  prediction. 

b.  Summary: 

(1)  The  current  state  of  the  art  for  dor¬ 
mant  reliability  prediction  is  not  directly 
applicable  to  the  OT&E  environment.  Most 
existing  methods  are  oriented  toward  the 
development  contractor's  or  SPO's  needs  and 
are  concerned  with  developing,  predicting, 
and  validating  accomplishment  of  specifica¬ 
tion  requirements.  It  is  rare  to  find  dormant 
reliability  requirements  explicitly  stated  in 
user  requirements  documents. 

(2)  The  current  methodologies  for  project¬ 
ing  weapon  systems  dormant  reliability  early 
in  the  system’s  life  cycle  tend  to  focus  at  the 
part  level.  Prediction  techniques  rely  on 
parts  count  and  part  stress  analysis  methods. 
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The  expanding  use  of  K  factors  to  adjust  or 
modify  failure  rates  to  account  for  varying 
stresses  imposed  by  different  applications  and 
environments  also  seems  to  be  more  appro¬ 
priate  at  the  part  level.  Applicability  at  the 
system  or  even  the  major  subsystem  level 
requires  further  study  and  development. 
Accelerated  testing  techniques  are  also  tai¬ 
lored  to  part  testing.  In  fact,  it  is  doubtful 
that  accelerated  testing  could  be  effectively 
applied  at  the  system  level  without  signifi¬ 
cant  additional  study  and  analysis.  Surveil¬ 
lance  testing  can  also  provide  potentially 
useful  data,  but  it  will  generally  come  from 
a  similar  program.  Modeling/simulation  can 
be  used  to  investigate  the  dormant  reliability 
impact  on  operational  availability. 

(3)  There  are  many  sources  of  useful  data 
for  the  operational  tester  to  use  when  devel¬ 
oping  early  predictions  of  a  new  system’s 
operational  reliability.  However,  extreme 
caution  must  be  exercised  to  ensure  that  the 
nature  of  the  data  (e.g.,  source,  derivation, 
similarities/dissimilarities,  etc.)  is  thoroughly 
understood. 

A2-6.  Tools  and  Techniques: 

a.  Availability  and  Applicability.  Tools 
and  techniques  for  estimating,  predicting, 
projecting,  or  assessing  dormant  reliability 
during  OT&E  are  limited  in  terms  of  their 
availability  for  use  and  their  direct  appli¬ 
cability  in  terms  of  the  inherent  limitations. 
Given  this  situation,  select  the  best  available, 
modify  where  possible,  or  suggest  new  meth¬ 
ods  to  pursue  in  the  future.  Requirements 
exist  for  tools  and  techniques  to  establish 
initial  dormant  reliability  estimates  and  to 
project  those  estimates  to  maturity.  Addi¬ 
tionally,  methods  to  determine  appropriate 
tests  to  obtain  dormant  reliability  assess¬ 
ment  data  are  needed. 

b.  Establishing  Initial  Estimates.  A 
number  of  techniques  for  establishing  initial 
dormant  reliability  estimates  exist.  Most  of 
these  techniques  are  applicable  at  the  piece- 
part— rather  than  the  system — level,  base  the 
estimate  on  comparable  systems,  or  use 
adjustment  factors.  Discussions  of  some  of 
the  most  frequently  used  techniques  follow: 

(1)  MIL-HDBK  217E,  Reliability  Predic¬ 
tion  of  Electronic  Equipment: 

(a)  MIL-HDBK  217E  contains  the  most 
widely  used  methods  for  initial  estimates  of 
reliability.  It  applies  mainly  to  electronic 
components  at  the  piecepart  level.  There 
are,  however,  some  considerations  which  may 
render  this  approach  useful  for  projection 
purposes. 

(b)  Although  MIL-HDBK  217E  meth¬ 
ods  result  in  an  estimate  of  inherent  dor¬ 


mant  reliability  generally  not  totally  applica¬ 
ble  to  field  conditions,  system  OR  storage 
reliability  may  be  adequately  represented  b> 
this  estimate.  An  understanding  of  the 
system  under  evaluation  and  the  proposed 
maintenance  concept  will  allow  the  analyst 
to  determine  the  appropriateness  of  this 
method. 

(c)  The  MIL-HDBK  217E  technique 
may  be  useful  when  the  system  is  composed 
of  relatively  few  subsystems,  e.g.,  a  tactical 
missile.  Once  initial  estimates  of  dormant 
reliability  have  been  established  for  each 
subsystem,  a  system  estimate  can  be  built  up 
from  them. 

(2)  Pieceparts  for  Nonelectronics.  Part 
reliabilities  and  piecepart  methodologies  for 
other  than  electronic  devices  are  available 
in  Rome  Air  Development  Center  and  Army 
Missile  Command  handbooks.  The  parts 
tend  to  be  aggregated  at  higher  levels  than 
those  of  the  MIL-HDBK  217E  listings  (e.g., 
generators,  pumps,  actuators,  regulators, 
rocket  engines,  valves).  Thus,  a  buildup 
approach  is  further  simplified.  The  draw¬ 
backs  to  these  documents  are  that  they  have 
not  received  "MIL-STD"  status  and  are 
slightly  harder  to  procure.  They  are,  how¬ 
ever,  in  current  common  usage. 

(3)  Piecepart  Surrogates.  If  very  quick 
estimates  are  necessary,  there  are  several 
estimating  relationships  which  have  been 
used.  These  estimating  relationships  essen¬ 
tially  provide  substitutes  for  an  aggregated 
parts  level.  Substitute  measures  include 
such  things  as  complexity,  volume,  weight, 
functions,  and  cost.  Simple  or  weighted 
averages  can  also  be  substituted  for  an 
actual  parts  count.  Substitute  methods 
provide  only  a  gross  estimate  of  reliability 
and  are  generally  used  for  very  quick  trade¬ 
off  analyses  by  system  developers.  Documen¬ 
tation  on  these  methods  is  virtually  nonexis¬ 
tent. 

(4)  Comparability  Analysis: 

(a)  Comparability  analysis  is  a  suc¬ 
cessful  and  recognized  technique.  There  are, 
however,  some  constraints.  The  data  base 
used,  the  skill  of  the  person  making  the 
comparability  decisions,  various  adjustments 
that  must  be  made  when  100-percent  com¬ 
parable  data  do  not  exist,  and  the  ingenuity 
of  the  analyst  applying  the  results  are  all 
limiting  factors.  Even  with  these,  however, 
comparability  analysis  is  probably  the  single 
best  technique  that  can  be  used  when  firm 
reliability  values  based  on  actual  system 
experience  and  data  are  not  available. 

(b)  This  approach  requires  detailed 
knowledge  of  the  specific  system  under 
consideration  as  well  as  other  systems/subsys- 
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terns  which  may  be  similar.  Given  that 
comparable  or  similar  subsystems  exist,  the 
problem  then  becomes  one  of  homing  in  on 
dormant  reliability  by  subsystem.  If  data 
on  dormancy  for  the  similar  systems  do  not 
exist,  one  is  left  with  factoring  operating 
failures  to  dormant  failures,  and  this  process 
is  generally  not  very  accurate.  The  need  to 
collaborate  with  other  agencies  who  maintain 
dormant  data  bases  thus  becomes  imperative. 

(5)  K  Factors: 

(a)  K  factors  discussed  in  paragraph 
20.5  are  adjustments  from  one  condition  or 
set  of  conditions  to  other  conditions.  In  the 
general  sense,  the  term  adjustment  factor  is 
more  appropriate,  but  the  use  of  the  designa¬ 
tor  "K"  has  become  common  to  reliability 
engineering  and  reliability  handbooks. 

(b)  It  is  impossible  to  generalize  the 
applicability  or  validity  of  adjustment  factors. 
First,  the  number,  type,  and  application  of 
adjustment  factors  are  almost  limitless.  Sec¬ 
ond,  the  validity  depends  on  the  application 
itself.  The  use  of  adjustment  factors  tends 
to  be  more  art  than  science  and  is  therefore 
strongly  influenced  by  the  skill,  expertise, 
experience,  and  ingenuity  of  the  person  using 
them.  Indiscriminate  or  uninformed  use  of 
adjustment  factors  is  dangerous. 

(c)  Several  studies  attempt  to  adjust 
from  operating  to  nonoperating  conditions. 
This  particular  adjustment  appears  to  have 
very  limited  potential  application  unless  done 
at  the  piecepart  level.  Results  vary  widely 
and  there  appears  to  be  no  universal  applica¬ 
tion.  Use  of  a  system-  or  subsystem-level 
adjustment  for  operating  to  nonoperating 
failure  rates  should  be  used  only  as  a  last 
resort,  at  least  until  much  more  research 
has  been  accomplished. 

(6)  Other  Estimating  Techniques.  There 
are  several  additional  estimating  techniques 
which  have  potential  for  application  to  the 
assessment  of  dormant  reliability,  but  for  the 
most  part  they  are  not  proven  and  should  be 
viewed  only  as  possibilities.  These  tech¬ 
niques  are  based  on  adaptation  or  extrapola¬ 
tion  of  contractor  screening  and  testing. 
These  techniques  include  deriving  estimates 
of  dormant  reliability  from  acceptance  tests, 
accelerated  stress  and  temperature  tests, 
item  shelf  life,  and  power-on/power-off  cy¬ 
cling. 

c.  Projections  to  Maturity: 

(1)  Projection  methodologies  are  general¬ 
ly  applicable  to  dormant  reliability  as  long 
as  the  nature  of  dormancy  is  carefully  con¬ 
sidered  as  part  of  the  process. 

(2)  In  the  ideal  case,  inherent  dormant 
reliability,  OR  storage  reliability,  and  dor¬ 
mant  reliability  by  state  (e.g.,  transport) 


should  be  closely  monitored  from  the  first 
estimates  and  specifications  forward  to  essen¬ 
tially  plot  a  bathtub  curve.  The  nature  of 
dormancy  and  the  weapon  system  develop¬ 
ment  process  are  such  that  one  would  expect 
to  see  reliability  changes  appear  in  jumps  or 
by  lot.  Thus,  reliability  growth  curves 
should  look  more  like  step  functions  than 
smooth  curves. 

(3)  Another  aspect  of  the  nature  of  dor¬ 
mancy  that  impacts  projections  relates  to  the 
definition  itself.  Dormancy  is  defined  at  the 
system  level  and  permits  only  on-equipment 
maintenance  and  BIT  or  functional  checks 
necessary  to  maintain  the  desired  status.  As 
a  weapon  system  matures  and  requires 
maintenance  for  dormant  failures,  "old"  parts 
and  subsystems  will  probably  be  used  for 
repair.  These  parts  and  subsystems  may  be 
experiencing  age-related  failures  themselves. 

One  might  see  an  artificially  early  wearout 
of  the  system  when  repair  is  performed  using 
parts  and  subsystems  that  are  more  likely  to 
fail  because  of  theii  own  age. 
d.  Testing  for  Dormant  Reliability: 

(1)  The  nature  of  the  dormancy  problem 

is  such  that  traditional  test  means  and 
methodologies  often  do  not  apply.  Individual  \ 

subsystems  of  a  given  weapon  system  may 

not  be  amenable  to  all  tests  for  the  effects  of 
dormancy.  For  example,  power-on/power-off 
tests  on  rocket  motors  and  warheads  are  not 
appropriate,  whereas  they  would  be  appropri¬ 
ate  for  guidance  units. 

(2)  From  the  OT&E  perspective,  it  is 
critically  important  that  AFOTEC  pursues 
early  involvement  and  collaboration  in  the 
development/acquisition  process.  A  real  issue 
centers  around  deciding  which  tests  are 
appropriate  and  necessary  for  AFOTEC  to 
conduct  to  assess  system  dormant  reliability. 

(3)  The  process  involved  in  this  decision 

is  based  on  a  straightforward  and  logical 
progression.  First,  early  detailed  knowledge 
of  the  system  itself  is  necessary.  Second,  a 
review  of  the  maintenance  and  operational 
concepts  should  be  performed.  If  these 
concepts  are  not  available,  early  collaboration 
with  the  development/user  community  will 
help  in  providing  at  least  provisional  con¬ 
cepts.  From  these  concepts,  a  life  cycle 
profile  should  be  constructed  to  define  hours 
the  system  is  exposed  to  in  each  of  its  envi¬ 
ronments.  Based  on  the  life  cycle  profile, 
decisions  can  be  made  regarding  the  states  a 

of  dormancy  which  are  sensitive  to  OT&E  fl 

and  appropriate  tests  by  system/subsystem. 

In  addition,  close  monitoring  of  engineering 
change  proposals  and  engineering  fixes 
should  provide  an  input  into  the  decision¬ 
making  process. 
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A2-7.  Dormant  Reliability  Assessment 
Approach: 

a.  Advance  Planning: 

(1)  Review  program  documentation  from 
the  SPO  and  contractor  for  critical  issues  and 
areas  of  risk  associated  with  dormant  relia¬ 
bility. 

(2)  Review  operational  and  maintenance 
concepts  for  background. 

(3)  Ensure  that  the  failure  definitions  to 
be  applicable  throughout  testing  are  incor¬ 
porated  in  appropriate  program  documenta¬ 
tion,  e.g.,  PMD,  TEMP,  TPO,  contract,  etc. 

(4)  Formulate  the  life  cycle  profile  model. 

(5)  Ensure  that  a  piecepart  dormant 
reliability  prediction  is  accomplished  by  the 
SPO  or  contractor.  Review  the  prediction  if 
one  is  already  accomplished. 

(6)  Review  system  design  for  dormant 
reliability  considerations. 

(7)  From  initial  reliability  predictions, 
determine  preliminary  number  of  assets  for 
system  surveillance  tests  during  OT&E  and 
recommend  that  the  assets  be  incorporated 
into  the  contract  and  SPO  budget. 

(8)  Review  the  contract  to  determine  if 
failure  analyses  are  required.  If  failure 
analyses  are  not  included,  recommend  con¬ 
tract  amendment  to  include  analyses. 

(9)  Review  the  contract  and  DT&E  test 
plan  for  contractor-accelerated  testing  of 
pieceparts  and  subsystems.  If  accelerated 
testing  is  not  included,  recommend  contract 
addition. 

b.  Detailed  Test  Planning: 

(1)  Review  the  current  design  and  the 
current  operations  and  maintenance  concepts 
to  update  the  life  cycle  profile,  OT&E  objec¬ 
tives,  and  test  methods. 

(2)  Review  failure  analyses. 

(3)  Update  the  reliability  prediction 
because  of  design  or  operation  and  main¬ 
tenance  concept  changes. 

(4)  Review  the  failure  modes,  effects,  and 
criticality  analysis  (FMECA)  for  updating 
OT&E  test  plan  objectives  and  methodology. 

(5)  Recommend  initiation  of  surveillance 
tests  of  subsystems  and  components. 

(6)  Accomplish  detailed  test  methodology 
for  appropriate  system  tests  to  address 
sensitive  areas. 

(7)  Ensure  that  assets  are  identified  for 
above  tests. 

c.  OT&E: 

(1)  Review  current  design  and  current 
operational  and  maintenance  concepts. 

(2)  Update  the  life  cycle  profile,  as  neces¬ 
sary. 

(3)  Review  failure  analyses. 

(4)  Update  dormant  reliability  prediction 
for  changes  in  system  design  and/or  opera¬ 


tional  and  maintenance  concept  changes. 

(.5)  Compare  DT&E  and  early  componer.: 
and  subsystem  accelerated  and  surveillance 
test  data  to  preliminary  OT&E  data. 

(6)  Document  deficiencies  and  review 
engineering  fixes  of  the  deficiencies. 

(7)  Compare  test  data  for  failure  rates  to 
scheduled  inspection  period. 

(8)  Review  and  analyze  test  equipment 
effectiveness. 

d.  FOT&E: 

(1)  Refine  life  cycle  profile,  as  necessary. 

(2)  Develop  FOT&E  objectives  if  differ¬ 
ent  from  OT&E  objectives. 

(3)  Establish  new  MOEs  if  necessary. 

(4)  Continue  surveillance  tests. 

(5)  Update  failure  rates  and  compare  to 
predicted  failure  rate. 

(6)  Analyze  and  compare  OT&E  and 
production  failure  rates  for  statistical  differ¬ 
ences. 

(7)  Analyze  DT&E  (accelerated  and 
surveillance  tests)  and  production  failure 
rates  for  statistical  differences. 

A2-8.  Lessons  Learned: 

a.  The  notion  of  dormancy  is  well  estab¬ 
lished,  and  clear  definitions  of  dormant- 
related  terminology  within  the  context  of  the 
specific  system  being  developed  are  essential 
to  properly  estimate  and  test  for  dormant 
system  reliability. 

b.  Reliability  prediction  through  analytical 
estimation  and  test  techniques  is  possible. 
Although  some  areas  have  not  been  vali¬ 
dated,  they  look  promising.  Both  types  of 
techniques  cover  the  total  spectrum  of  dor¬ 
mant  reliability  assessment.  Therefore,  begin 
by  updating  initial  contractor  estimates  and 
transition  to  testing  when  practical. 

c.  Early  formulation  of  the  system’s  life 
cycle  profile  model  is  essential  to  the  entire 
process  of  defining  dormancy,  structuring  a 
comprehensive  test  program,  and  assessing 
the  effects  of  dormancy  on  operational  relia¬ 
bility  and  logistics  reliability. 

d.  Reliability  prediction  based  on  the 
piecepart  count  method  should  be  considered. 
During  the  early  planning  phase,  it  may 
provide  the  only  source  of  available  data. 

e.  Early  involvement  during  the  conceptual 
phase  of  the  system  acquisition  cycle  is 
essential. 

f.  A  critical  element  of  "dormant  failure 
rate"  for  systems  maintained  under  a  wooden 
round  concept  appears  to  be  induced  failures. 
System  reliability  can  be  improved  with  a 
periodic  test  concept,  provided  the  induced 
failure  rate  can  be  held  to  a  low  level. 

g.  Dormancy  was  defined  as  "Those  states 
in  which  a  system  is  not  operating  or  is 
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being  maintained  in  OR  storage — including 
all  on-equipment  maintenance....''  Since 
dormancy  is  not  normally  defined  at  the 
subsystem  or  lower  level,  parts  and  subsys¬ 
tems  in  storage  are  not  included  in  the  aging 
of  the  system. 

h.  Dormancy  is  defined  at  the  system  level 


and  permits  only  on-equipment  maintenance 
and  BIT  or  functional  checks  necessary  to 
maintain  the  desired  status.  Therefore,  parts 
and  subsystems  that  are  in  storage  are  net 
considered  subject  to  the  effects  of  dormancy 
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AVAILABILITY  IN  AIRCRAFT,  AVIONICS,  SIMULATORS, 
GROUND  COMMUNICATION/WEATHER  SYSTEMS,  AND  MUNITIONS 


A3-1.  Discussion: 

a.  Availability  assessments  of  aircraft, 
avionics  systems,  munitions,  simulators, 
ground  communication,  and  weather  systems 
differ  in  degree.  In  all  cases,  the  assessment 
must  begin  with  a  clear  understanding  of  the 
mission  of  the  system,  its  critical  components, 
its  intended  operational  environment,  and  its 
maintenance  concept.  From  this  baseline,  a 
test  concept  may  be  developed. 

b.  For  aircraft  assessments,  the  entire 
aircraft  with  all  its  systems  and  subsystems 
is  assessed  in  terms  of  its  availability. 
Systems  and  subsystems  are  important  only 
to  the  extent  that  they  contribute  to  the 
mission  and  support  of  the  aircraft. 

c.  For  avionics  system  assessments,  the 
avionics  system  must  perform  its  mission  as 
part  of  the  larger  aircraft  system.  If  the 
avionics  system  is  critical  to  the  mission  of 
the  aircraft,  the  impact  of  the  avionics  sys¬ 
tem’s  availability  on  the  availability  of  the 
host  aircraft  must  be  assessed.  If  the  avion¬ 
ics  system  is  not  critical  to  the  mission  of 
the  aircraft,  no  availability  assessment  is 
required  (nor  is  it  desired). 

d.  Simulator  assessments  are  unique  in 
that  simulators  are  normally  procured  in 
limited  quantities  and  are  not  generally 
controlled  by  a  production  decision.  These 
assessments  are  usually  accomplished  in 
conjunction  with  an  AFSC-conducted  qualifi¬ 
cation  test  as  a  combined  QT&E/QOT&E 
IAW  AFR  80-14.  Generally,  simulator  as¬ 
sessments  have  two  phases  (in-plant  and  on¬ 
site)  each  with  a  separate  "quick-look"  report. 
Additionally,  a  maintainability  demonstration 
is  normally  conducted  during  the  on-site 
phase. 

e.  Ground  communication/weather  systems 
differ  from  aircraft  and  avionics  in  that  they 
are  generally  required  to  operate  24  hours 
per  day.  These  systems  are  frequently 
deployed  to  remote  locations,  thus  adding  a 
significant  amount  of  transportation  time  to 
any  maintenance  requirement  Environment¬ 
al  factors  such  as  weather  require  increased 
consideration  during  test  planning.  Technol¬ 
ogy  has  developed  to  the  point  where  these 
systems  may  continue  to  fulfill  mission 
requirements  with  significant  portions  inoper¬ 
able.  Finally,  repair  of  these  systems  can 
frequently  be  performed  while  they  continue 
to  operate.  All  of  these  differences  combine 
to  change  specific  portions  of  the  availability 
assessment.  When  C’l  systems  are  part  of 
an  aircraft,  they  are  treated  as  avionics. 


f.  Munitions  and  missiles  include  a  wide 
variety  of  systems  that  can  be  used  for  a 
number  of  different  missions  and  range  in 
complexity  from  ammunition  to  cruise  mis¬ 
siles.  In  general,  missiles  and  munitions 
must  be  able  to  withstand  long  periods  of 
storage  with  little  maintenance  and  still 
perform  with  high  reliability.  Missiles  differ 
from  munitions  in  having  a  self-contained 
propulsion  system  to  move  from  the  launch 
point  to  the  target. 

A3-2.  Availability  Test  Objectives: 

a.  The  following  paragraphs  discuss  the 
availability  objectives  AFOTEC  uses  for 
aircraft,  avionics,  simulators,  ground  com¬ 
munication/weather  systems,  and  munitions. 
The  structure  of  the  test  plan  and  the  phras¬ 
ing  of  each  objective  will  vary  depending  on 
the  system  being  assessed,  the  type  of  OT&E 
conducted,  and  statement  of  the  operational 
requirement  for  availability. 

b.  The  goal  of  availability  assessment  is 
to  determine  if  the  system  can  meet  the 
user’s  specified  availability  requirements. 
Availability  is  a  measure  of  the  degree  to 
which  an  item  is  in  an  operable  and  com- 
mittable  state  at  the  start  of  a  mission  when 
the  mission  is  called  for  at  a  random  point 
in  time.  Availability  is  a  function  of  the 
system’s  reliability  and  maintainability  char¬ 
acteristics  and  on  its  logistics  supportability. 
Measures  of  availability  are  generally  ex¬ 
pressed  as  percentages,  e.g.,  either  the  per¬ 
centage  of  times  a  system  is  capable  of 
performing  its  mission  or  the  percentage  of 
time  a  fleet  is  capable  of  performing  its 
mission.  During  IOT&E,  little  or  no  mean¬ 
ingful  availability  data  may  be  available. 
Under  these  circumstances,  the  emphasis  of 
the  availability  assessment  will  shift  from 
quantitative  to  qualitative.  There  are  some 
cases  where  data  from  the  combined  DT&E/ 
IOT&E  test  period  may  be  used  to  help 
evaluate  availability. 

c.  Examples  of  aircraft  (F-X)  availability 
objectives  are: 

(1)  Assess/evaluate  the  F-X  operational 
availability. 

(2)  Assess/evaluate  the  F-X  operational 
availability  in  support  of  mission  Y. 

d.  Examples  of  avionics  system  (AV-1) 
availability  objectives  are: 

(1)  Assess/evaluate  the  AV-1  operational 
availability  as  it  affects  readiness  for  mission 
X. 

(2)  Assess/evaluate  the  AV-1  operational 
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availability  as  it  affects  readiness  for  mission 
X  when  installed  on  host  system  Y. 

Note:  If  AV-1  is  not  required  for  any  mis¬ 
sion  (either  as  stand-alone  or  when  installed 
on  a  host  system),  there  should  be  no  avail¬ 
ability  objective  started  in  the  test  plan. 
The  reason  for  this  is  availability  has  little 
operational  meaning. 

e.  Example  of  simulator  availability  objec¬ 
tive:  Assess/evaluate  simulator  X  avail¬ 

ability. 

f.  Example  of  ground  communication/ 
weather  system  availability  objective:  As¬ 
sess/evaluate  system  X  operational  availabil¬ 
ity- 

g.  Example  of  munitions  availability 
objective:  Assess/evaluate  munition  X  opera¬ 
tional  availability. 

A3-3.  Availability  Measures  of  Effective¬ 
ness: 

a.  General.  The  selection  of  MOEs  for 
availability  should  be  based  on  the  terms 
found  in  AFP  57-9  and  the  ORD.  These 
MOEs  are  extremely  difficult  to  measure 
during  IOT&E  where  many  support  elements 
are  not  representative  of  the  operational 
environment.  During  a  typical  IOT&E, 
extensive  modeling  is  required  to  estimate 
and  project  these  availability  measures. 
During  FOT&E,  with  representative  support 
elements  in  an  operational  organization, 
these  MOEs  can  be  measured  directly. 

b.  Aircraft  and  Avionic  System  Avail¬ 
ability  MOEs: 

(1)  Sortie  Generation  Rate  (SGR).  SGR 
is  the  number  of  sorties  that  can  be  flown 
per  aircraft  per  day  under  specified  opera¬ 
tional  and  maintenance  concepts.  A  sortie 
starts  when  the  aircraft  begins  to  move 
forward  on  takeoff  and  ends  when  the  air¬ 
craft  returns  to  the  surface  and  either  the 
engines  are  stopped,  the  aircraft  is  on  the 
surface  for  5  minutes  (whichever  occurs  first), 
or  a  change  is  made  in  the  crew.  SGR  is 
calculated  through  direct  measurement  or 
from  simulation  results  depending  on  the 
realism  of  the  test  environment.  Because 
most  aircraft  can  perform  several  missions, 
it  may  be  necessary  to  calculate  several 
SGRs  to  represent  the  full  range  of  opera¬ 
tional  mission  requirements.  The  following 
formula  applies: 

total  number  of  sorties/ 

SGR  _  total  number  of  aircraft 
number  of  days 

This  measure  typically  applies  to  the  total 
aircraft  system  and  is  not  used  for  avionics 
systems  and  simulators.  However,  the  im¬ 


pact  of  a  new  avionics  system  on  the  host 
aircraft’s  SGR  may  be  measured. 

(2)  Fully  Mission  Capable  'FMC)  Rate. 
The  FMC  rate  is  the  percentage  of  possessed 
hours  that  a  system  is  capable  of  performing 
all  its  assigned  peacetime  and  wartime 
missions.  The  critical  identification  of  compo¬ 
nents  is  required  for  the  missions  to  deter¬ 
mine  the  status  of  the  aircraft.  The  follow¬ 
ing  formula  applies: 

FMC  =  number  of  hours  FMC 
possessed  hours 

This  MOE  has  the  advantage  of  being  a  well- 
established  parameter,  with  the  data  nec¬ 
essary  to  calculate  it  generally  available 
through  automated  systems  in  the  operation¬ 
al  environment.  However,  the  FMC  rate 
should  not  be  directly  measured  during 
IOT&E  because  support  elements  will  not  be 
representative  of  the  intended  operational 
environment.  Instead,  simulation  should  be 
used  as  a  basis  for  calculating  the  FMC  rate 
during  IOT&E. 

(3)  Partially  Mission  Capable  (PMC)  Rate. 
The  PMC  rate  is  the  percentage  of  possessed 
hours  that  a  system  is  capable  of  performing 
at  least  one,  but  not  all,  of  its  assigned 
peacetime  and  wartime  missions.  The  criti¬ 
cal  identification  of  components  is  required 
for  the  missions  to  determine  the  status  of 
the  aircraft.  For  analysis  purposes,  the  PMC 
rate  may  be  divided  into  categories  as  fol¬ 
lows:  PMC  for  maintenance  (PMCM),  PMC 
for  supply  (PMCS),  and  PMC  for  both  main¬ 
tenance  and  supply  (PMCB).  The  following 
formula  applies: 

Note:  Care  must  be  taken  if  PMC  is  divided 
out  because  of  the  difficulty  in  separating  the 
data  during  test. 

PMC  =  number  of  hours  PMC 
possessed  hours 

As  before,  the  PMC  rate  should  not  be  direct¬ 
ly  measured  during  IOT&E.  Instead,  simula¬ 
tion  should  be  used  as  a  basis  for  calculating 
the  PMC  rate  during  IOT&E. 

(4)  Mission  Capable  (MC)  Rate.  The  MC 
rate  is  the  percentage  of  possessed  hours  a 
system  is  capable  of  performing  any  or  all 
of  its  assigned  missions.  The  following 
formula  applies: 

number  of  FMC  hours  +  number 

MC  = _ of  PMC  hours _ 

possessed  hours 

The  MC  rate  should  not  be  directly  measured 
during  IOT&E.  Simulation  should  then  be 
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used  as  a  basis  for  calculating  the  MC  rate. 

c.  Simulator  Availability  MOE.  For 
simulators,  overall  availability  has  little 
meaning.  Because  simulators  are  scheduled 
and  used  by  individual  mission  categories, 
availability  for  each  mission  category  must 
be  measured  and  reported.  A  typical  meas¬ 
ure  is  the  ratio  of  successful  missions  to 
scheduled  missions.  The  definition  of  what 
"successful"  means  should  be  included  in  the 
test  plan.  A  weighted  availability  rate  may 
be  calculated  by  using  planned  simulator 
utilization  rates. 

cL  Communication/Weather  System 
Availability  MOE.  Uptime  ratio  (UR)  is 
the  most  common  measure  of  availability  for 
ground  communication/weather  systems.  It 
is  the  equivalent  of  mission-capable  rate  for 
aircraft  and  avionics  systems.  However, 
since  possessed  time  equals  planned  operat¬ 
ing  time  for  Cl  systems,  an  additional 
formula  may  be  used  in  its  calculation.  A 
word  of  caution:  since  the  IOT&E  environ¬ 
ment  is  seldom  typical  of  the  operational 
environment,  the  MDT  portion  of  the  first 
formula  should  be  derived  by  simulation  and 
not  measured  directly.  The  following  form¬ 
ulas  may  be  used  to  calculate  UR: 

Ur  _ _ uptime 

uptime  +  downtime 

UR  —  MTBCF 

MTBCF  +  MDT 

Other  formulas  are  defined  in  AFP  57-9,  but 
these  show  the  relationships  involved, 
e.  Munition  Availability  MOEs: 

(1)  Sortie  Surge  Generation.  Sortie  surge 
generation  is  the  number  of  operable  muni¬ 
tions/missiles  that  can  be  assembled,  deliv¬ 
ered,  and  loaded  to  meet  wartime  sortie  re¬ 
quirements.  The  aircraft  type;  quantities  of 
personnel,  aircraft,  support  equipment,  and 
munitions;  and  time  constraints  are  based  on 
operational  requirements  and  should  be 
established  by  the  using  command.  Sortie 
surge  generation  can  be  calculated  through 
direct  measurement  but  is  normally  esti¬ 
mated  using  a  simulation  model  because  of 
the  lack  of  assets  to  conduct  a  representative 
generation  during  OT&E. 

(2)  Mission-Capable  (MC)  Rate.  MC  rate 
is  used  only  for  systems  that  can  be  tracked 
using  AFR  65-110  or  similar  reporting  sys¬ 
tems.  Many  missiles/munitions  are  tracked 
by  inventory  reporting  systems;  hence,  this 
term  may  not  apply.  In  cases  where  an  MC 
requirement  is  stated,  the  following  formula 
is  used: 


MC  =  FMC  +  PMC,  where 

_  number  of  hours  FMC 
possessed  hours 

and 

_  number  of  hours  PMC 
possessed  hours 


(3)  Asset/Stockpile  Availability.  A  base- 
level  asset/stockpile  availability  (A.;  is  the 
radio  of  the  number  of  munitions  on-hand  to 
the  number  of  munitions  assigned.  The 
number  of  munitions  on-hand  normally 
excludes  those  on-hand  assets  which  are  a 
disassembled  for  storage  and/or  testing  (i.e.. 
disassembled  munition  is  normally  not  an 
available  munition.  Once  assembled  and 
checked  out,  it  becomes  available;.  Equiva¬ 
lent  definitions  are  used  for  theater-level 
stockpile  availability  and  force-level  stockpile 
availability.  The  following  general  formula 
applies: 


A.  = 

a 


#  of  available  munitions 
total  munitions  in  inventory 


A3-4.  Availability  Data  Requirements: 

a.  Data  elements  required  to  calculate  the 
above  MOEs  vary  with  the  test  environment. 

b.  During  IOT&E,  the  test  environment 
usually  does  not  represent  the  intended 
operational  environment.  As  a  result,  direct 
measurement  (demonstrated  availability; 
cannot  be  used  to  produce  operationally 
meaningful  estimates  of  availability.  Instead, 
data  elements  are  collected  from  similar 
systems,  testing,  and  contractor  estimates. 
These  data  elements  are  then  used  in  simu¬ 
lation  models  to  estimate  operational  availa¬ 
bility.  The  following  are  examples  of  these 
data  elements: 

(1)  Mean  time  between  maintenance 
(MTBM). 

(2)  Mean  time  between  critical  failures 
(MTBCF). 

(3)  Mean  repair  time  (MRT). 

(4)  Mean  downtime  (MDT). 

(5)  Logistics  delay  times. 

(6)  Administrative  delay  times. 

(7)  Operational  mission  scenarios/mini¬ 
mum  essential  subsystem  lists  (MESL). 

(8)  Expected  manpower. 

(9)  Expected  sparing. 

(10)  Expected  support  equipment. 

(11)  Number  of  each  type  of  mission 
attempted  (simulators). 

(12)  Number  of  each  type  of  mission 
completed  (simulators). 

(13)  Environmental  factors  (ground  com- 
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munication/weather  systems). 

(14)  Expected  inventory  size/authorization 
and  delivery  schedules  (munitions). 

c.  During  FOT&E,  the  test  environment 
is  generally  representative  of  the  intended 
operational  environment.  Therefore,  the  data 
elements  listed  below  may  be  used  directly 
to  calculate  the  MOEs: 

(1)  EMC  hours,  PMC  hours,  and  NMC 
hours. 

(2)  Number  of  sorties. 

(3)  Number  and  type  of  assigned  aircraft, 
personnel,  and  SE. 

(.4)  Number  of  possessed  hours/days. 

(5)  Uptime  hours,  downtime  hours 
(ground  communication/weather  systems). 

(6)  Inventory  size  (munitions). 

d.  As  with  data  elements,  data  sources 
vary  with  the  test  environment. 

(1)  During  IOT&E,  the  data  elements  are 
available  from: 


■  a;  Contractor  reports, 
ib;  Data  from  other  OT&E  objective.- 
(e.g.,  reliability  and  maintainability’. 

(c)  Operational  and  maintenance  con¬ 
cepts. 

'd)  Weapon  system  and  equipment 
support  analysis  <’ WSESA)  or  logistics  support 
analyses  <LSA) 

(e)  System  effectiveness  data  svstem 
(SEDS). 

(f)  Similar  systems  in  test  or  deployed. 
(2)  During  FOT&E,  the  data  elements  are 

available  from: 

(a)  AFR  65-110,  Aerospace  Vehicle  unci 
Equipment  Inventory,  Status,  and  Utilization 
Reporting  System  (AVISURS). 

(b)  Maintenance  management  informa¬ 
tion  and  control  system  (MMICS). 

(c)  Status  data  from  OT&E  logbooks. 

(d)  Automated  data  systems  (CAMS. 
Micro  Omnivore,  etc.). 


( 
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SPACE  SYSTEMS  ASSESSMENT 


A4-1.  Discussion.  An  operational  test  and 
evaluation  for  a  space  system  includes  the 
projection  of  its  operational  suitability  and 
how  well  the  ground  segment  can  be  op¬ 
erated  and  maintained  by  military  or  con¬ 
tract  or  contractor  personnel  in  the  operating 
environment.  This  includes  identifying  those 
areas  where  improvements  will  be  desirable 
as  early  in  the  system’s  development  or  life 
as  possible.  Factors  included  in  the  evalua¬ 
tion  are  the  space  system’s  availability, 
reliability,  maintainability,  and  logistics 
supportability.  Where  applicable,  the  same 
factors  may  be  assessed  for  subsystems  to 
highlight  the  system’s  suitability  performance 
in  specific  areas. 

a.  General  Space  System  Segments. 
A  space  system  is  generally  divided  into 
three  segments  for  planning  and  operational 
purposes:  control,  mission,  and  space.  They 
are  defined  as: 

(1)  Control  Segment.  The  related  hard¬ 
ware,  software,  and  procedures  necessary  for 
command  and  control  of  a  satellite’s,  space¬ 
craft’s,  or  space  system’s  health  and  status. 

(2)  Mission  Segment.  Hardware,  soft¬ 
ware,  procedures,  and/or  data  needed  to 
utilize  the  payload  of  a  satellite,  spacecraft, 
or  space  system.  This  segment  may  provide 
communications  between  the  payload  and  the 
user,  data  acquisition  from  the  spacecraft, 
and  data  processing  and  data  transfer  to  the 
users.  These  space- to-ground  links  may  be 
characterized  by  the  requirement  for  extreme¬ 
ly  high  levels  of  availability  frequently 
achieved  through  redundancy.  This  segment 
may  include  user  control  of  the  mission 
segment. 

(3)  Space  Segment.  That  portion  of  a 
space  system  which  is  intended  to  operate 
in  space  including  the  associated  program- 
related  elements  of  the  launch  subsystem  (if 
applicable). 

(a)  The  space  segment  includes  the 
units  which  will  be  deployed  in  space  and 
the  vehicle  used  to  deploy  it.  This  is  the 
flight  hardware  to  which  traditional  suitabil¬ 
ity  elements  are  more  applicable.  The  limit¬ 
ed  inventory  and  activity  levels  currently 
experienced  in  the  space  segment  have  gener¬ 
ated  the  requirement  for  almost  total  contrac¬ 
tor  preparation  and  maintenance  of  the 
system.  Operationally  reusable/recoverable 
systems  may  require  Air  Force  maintenance 
and  logistics  support  and  will  require  a  more 
classic  approach  to  operational  suitability 
evaluation. 

(b)  Space  segments  for  the  most  part 


involve  procurement  of  a  few  items  over  an 
extended  period  of  time.  Frequently,  procure¬ 
ment  and  management  techniques  used  to 
acquire  space  segments  are  based  on  mission 
success  or  on-orbit  performance  incentive 
contracts.  These  acquisition  methods  limit 
hands-on  participation  by  DOD  personnel  in 
contractor  development  and  testing. 

(c)  Since  space  segment  testing  is 
normally  accomplished  by  contractor  per¬ 
sonnel,  DT&E  and  OT&E  test  objectives  and 
data  requirements  must  be  defined  as  early 
as  possible  and  provided  to  the  SPO  prior  to 
contract  initiation. 

b.  Space  Program  Management.  In 

addition  to  being  the  implementing  agency. 
AFSC’s  Space  Systems  Division  (SSD)  often 
assumes  the  role  of  using,  operating,  or 
supporting  agency.  Additionally,  some  space 
systems  do  not  follow  standard  weapon 
system  acquisition  processes.  For  example, 
frequently  there  is  no  production  decision. 
For  these  reasons,  test  policy  may  have  to  be 
tailored  to  meet  USAF  and  DOD  directives. 

c.  Suitability  OT&E  of  Space  Systems: 

(1)  An  increasing  level  of  Air  Force 
activity  in  space  is  anticipated,  with  the 
consequent  need  for  increasing  levels  of  Air 
Force  maintenance  and  logistics  support.  In 
addition  to  the  increasing  responsibilities  of 
the  Air  Force  involvement  in  space  regarding 
these  suitability  issues,  projected  plans  to 
deploy  systems  and  crews  to  space  on  an 
operational  basis  will  generate  the  type  of 
suitability  concerns  traditionally  addressed  in 
OT&E. 

(2)  Because  of  the  highly  specialized 
nature  of  past  space  activities,  most  current 
systems  support  is  contractor-furnished.  The 
development  of  future  Air  Force  operational 
space  systems  is  expected  to  require  in¬ 
creased  Air  Force  operational,  maintenance, 
and  logistics  support.  The  level  of  Air  Force 
involvement  will  be  a  prime  consideration  in 
developing  test  objectives  and  MOEs  from 
critical  issues. 

(3)  Although  there  are  many  problem 
areas  involved  in  conducting  an  operational 
suitability  evaluation  of  a  space  system,  early 
recognition  of  them  and  adequate  planning 
will  reduce  their  impact.  The  complexity  of 
each  of  the  segments  listed  above  and  the 
degree  of  Air  Force  participation  in  various 
segments  will  generate  different  critical 
issues,  objectives,  and  MOEs  for  each  seg¬ 
ment.  OT&E  of  each  segment  may  be  con¬ 
ducted  as  a  separate  evaluation.  Testing  of 
the  control  segment  is  similar  to  ground 
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communication  systems. 

A4-2.  Integration  of  Segments  and  Ob¬ 
jectives  into  Total  System  Evaluation: 

a.  Because  space  system  evaluation  in¬ 
volves  the  assessment  of  the  control,  mis¬ 
sion,  and  space  segments  through  somewhat 
independent  objectives  and  MOEs,  the  total 
system  suitability  evaluation  must  somehow 
integrate  each  of  the  segment  assessments. 
This  process  requires  knowledge  of  how  the 
segments  interact  with  each  other,  what  the 
various  impacts  of  segment  downtime  will 
have  on  the  total  system,  and  the  impact  of 
various  time  delays  on  system  performance. 
For  example,  an  item  of  control  equipment 
may  have  a  99-percent  availability  rate  but 
is  so  critical  to  total  system  performance  that 
a  99.9-percent  availability  rate  is  required. 
Alternatively,  a  2-month  launch  preparation 


period  may  be  acceptable  because  redundant 
systems  are  sufficient  to  allow  that  deiay. 

b.  A  method  of  integrating  the  evalua¬ 
tion  of  each  segment  is  through  use  of  simu¬ 
lation  to  model  the  entire  system.  Suitability 
performance  parameters  generated  by  a 
model  are  the  total  system  availability  or 
readiness  during  given  operational  periods 
and  the  capability  of  failure-free  operational 
status  over  a  given  period. 

A4-3.  Evolution  of  Space  Systems  As¬ 
sessment.  AFOTEC  is  assessing  the  opera¬ 
tional  suitability  of  many  space  systems, 
nevertheless,  the  process  of  evaluating  space 
systems  is  still  in  its  infancy.  In  essence,  it 
is  a  combination  of  various  techniques  dis¬ 
cussed  in  Part  Three.  As  further  experience 
is  acquired  through  the  evaluation  of  space 
systems,  this  attachment  will  be  expanded. 
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REFERENCE  DOCUMENTATION 


A5-1.  Current  DOD  and  USAF  Policy 
Guidance.  This  attachment  lists  DOD  and 
Air  Force  policy  guidance  documentation 
relating  to  the  operational  suitability  evalua¬ 
tion  requirements  during  the  acquisition  of 
major  weapon  systems.  The  information  is 
subject  to  frequent  changes.  The  reader 
should  ensure  that  only  the  current  edition 
of  publications  is  used. 

A5-2.  Office  of  Management  and  Budget 
(OMB)  Direction: 

a.  In  April  1976,  the  OMB  published 
Circular  A- 109,  Major  Systems  Acquisitions. 
It  prescribed  policies  for  all  executive  branch 
agencies  involved  in  acquiring  major  systems. 
It  is  based  on  the  general  policy  that  federal 
agencies,  when  acquiring  major  systems,  will 
express  needs  and  program  objectives  in 
mission  terms  and  not  equipment  terms. 
The  stated  major  system  acquisition  manage¬ 
ment  objectives  were  that  each  agency  would: 

(1)  Tailor  an  acquisition  strategy  for  each 
program  that  encompasses  demonstration, 
test,  and  evaluation  criteria. 

(2)  Maintain  a  capability  to  assess  acqui¬ 
sition  cost,  schedule,  and  performance  experi¬ 
ence  against  predictions  and  provide  such 
assessments  at  key  decision  points. 

b.  As  a  result  of  OMB  Circular  A- 109,  the 
DOD  and  the  Air  Force  revised  their  acquisi¬ 
tion  policies. 

A5-3.  Department  of  Defense  Directives 
and  Instructions  (DODD/DODD: 

a.  DODD  5000.1,  Defense  Acquisition. 
This  directive  establishes  a  disciplined  man¬ 
agement  approach  for  acquiring  systems  and 
material  that  satisfy  the  operational  user’s 
needs.  The  policies  in  this  directive  es¬ 
tablish  a  disciplined  approach  for  integrating 
the  efforts  and  products  of  the  Department 
of  Defense’s  requirements  generation;  acquisi¬ 
tion  management;  and  planning,  program¬ 
ming,  and  budgeting  systems. 

b.  DODI  5000.2,  Defense  Acquisition 
Management  Policies  and  Procedures. 
This  instruction  establishes  an  integrated 
framework  for  translating  broadly  stated 
mission  needs  into  stable,  affordable  acquisi¬ 
tion  programs  that  meet  the  operational 
user’s  needs  and  can  be  sustained,  given 
projected  resource  constraints.  The  instruc¬ 
tion  also  establishes  an  event-oriented,  rigor¬ 
ous  management  process  for  acquiring  quality 
products  that  emphasize  effective  acquisition 
planning,  improved  communications  with 
users,  and  aggressive  risk  management  by 


both  government  and  industry. 

c.  DOD  5000.2-M,  Defense  Acquisition 
Documentation  and  Reports.  This  man¬ 
ual  contains  procedures  and  formats  used  to 
prepare  various  acquisition  category  I  and  II 
reports,  including  milestone  documentation, 
periodic  in-phase  status  reports,  and  statu¬ 
tory  certifications. 

A5-4.  Air  Force  Regulations  (AFR)  and 
Pamphlets  (AFP).  Numerous  Air  Force 
documents  provide  guidance  for  a  system’s 
acquisition.  Air  Force  documents  of  par¬ 
ticular  interest  in  relation  to  operational 
suitability  evaluation  are  AFRs  57-1,  66-14, 
80-14,  800-2,  800-8,  and  800-18.  Synopses  of 
these  documents  are  presented  below. 

a.  AFR  55-43,  Management  of  Opera¬ 
tional  Test  and  Evaluation.  AFR  55-43 
provides  overall  guidance  for  management  of 
a  test  and  evaluation  program.  It  outlines 
the  responsibilities  of  various  participating 
agencies  and  the  general  process  of  test 
planning  and  execution  and  describes  the 
development  of  test  objectives  and  evaluation 
criteria.  For  example,  it  contains  such 
subjects  as  required  test  documentation 
related  to  acquisition  milestones,  test  design 
for  individual  elements  of  operational  suit¬ 
ability,  test  report  formats,  and  resource 
management  forms.  AFOTEC  Supplement  1 
to  this  regulation  provides  additional  guid¬ 
ance  to  HQ  AFOTEC  and  AFOTEC  field 
units. 

b.  AFR  57-1,  Operational  Needs,  Re¬ 
quirements,  and  Concepts.  This  publi¬ 
cation  outlines  Air  Force  policies,  procedures, 
and  responsibilities  for  identifying,  process¬ 
ing,  and  approving  operational  requirements 
which  result  in  research,  development,  test, 
and  evaluation  (RDT&E)  or  procurement  ap¬ 
propriations.  AFR  57-1  describes  the  criteria, 
content,  format,  and  approval  process  for 
mission  need  statement  (MNS),  and  opera¬ 
tional  requirements  documents  (ORD).  It 
also  provides  procedures  for  preparing  a 
mission  need  statement  '"MNS),  processing 
and  coordinating  joint  service  operational 
requirements  (JSOR),  international  coop¬ 
erative  programs  including  North  Atlantic 
Treaty  Organization  (NATO)  and  North 
American  Aerospace  Defense  (NORAD)  Com¬ 
mand  program  within  the  context  of  the 
overall  acquisition  process. 

c.  AFP  57-9,  Defining  Logistics  Re¬ 
quirements  in  Statements  of  Opera¬ 
tional  Need.  Defines  procedures  and  out¬ 
lines  guidance  for  including  readiness  and 
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detailed  explanations  of  readiness-related 
terms  and  the  integrated  logistics  support 
elements  with  discussions  and  examples  of 
their  development  and  inclusion  in  MNSs. 

d.  AFR  65-110,  Aerospace  Vehicle  and 
Equipment  Inventory,  Status,  and  Utili¬ 
zation  Reporting  System  (AVTSURS). 
This  regulation  establishes  inventory,  status, 
and  utilization  reporting  policy  and  proce¬ 
dures  for  selected  aerospace  vehicles  and 
equipment. 

e.  AFR  66-1,  Maintenance  Manage¬ 
ment  Policy.  This  regulation  establishes 
the  maintenance  management  system  for  all 
Air  Force  and  Air  Reserve  Forces  activities 
which  maintain  aircraft,  missiles,  munitions, 
support  equipment,  avionics,  training  equip¬ 
ment,  and  communications  and  electronics 
equipment.  It  implements  the  provisions  of 
AFR  66-14  which  pertain  to  on-equipment 
and  off-equipment  maintenance. 

f.  AFR  66-14,  Equipment  Maintenance 
Policies,  Objectives,  and  Responsibilities. 
AFR  66-14  sets  up  the  principles  to  be  used 
in  developing  maintenance  concepts  and 
outlines  the  policies  and  procedures  for 
managing  the  Air  Force  equipment  and 
maintenance  program. 

g.  AFR  80-14,  Test  and  Evaluation: 
This  regulation  outlines  the  policy  and  proce¬ 
dures  for  managing  test  and  evaluation 
activities  during  the  development,  production, 
and  deployment  of  defense  systems  in  the  Air 
Force.  It  assigns  test  and  evaluation  respon¬ 
sibilities  to  the  implementing  command, 
AFOTEC,  and  the  operating  and  supporting 
commands.  AFR  80-14  states  that  OT&E  is 
conducted  in  as  operationally  representative 
condition  as  possible  to  estimate  or  to  refine 
estimates  of  a  system’s  operational  effective¬ 
ness  and  suitability  and  to  identify  operation¬ 
al  deficiencies  and  the  need  for  modifications. 

h.  AFR  800-2,  Acquisition  Program 
Management.  This  regulation  states  the 
policy  for  managing  all  Air  Force  acquisition 
and  modification  programs.  It  implements 
DODD  5000.1  and  DODI  5000.2,  establishes 
policy  on  acquisition  programs,  and  outlines 
participating  agency  responsibilities. 

i.  AFR  800-8,  Integrated  Logistics 
Support  (ILS)  Program.  This  regulation 
states  the  Air  Force  policy  for  ILS  manage¬ 
ment  and  defines  requirements  for  applying 
ILS  throughout  the  life  cycle  of  systems, 
equipment,  and  modification  programs. 

j.  AFR  800-14,  Life-Cycle  Management 
of  Computer  Resources  in  Systems. 
Establishes  policy  for  the  acquisition  and 
support  of  computer  equipment  and  computer 
programs  employed  as  dedicated  elements, 
subsystems,  or  components  of  systems  devel¬ 


oped  or  acquired  under  the  program  manage¬ 
ment  concept  established  in  AFR  S00-2. 

k.  AFR  800-18,  Air  Force  Reliability 
and  Maintainability  Program.  This 
regulation  establishes  the  policy  for  devel¬ 
oping,  acquiring,  maintaining,  modifying,  and 
operating  Air  Force  systems  that  are  relia¬ 
ble  and  maintainable.  Air  Force  R&M  policy 
pertains  to  any  action,  procedure,  technique, 
or  design  that  enhances  system  combat 
effectiveness  and  operational  suitability  by 
making  the  system  either  more  reliable  or 
easier  to  maintain,  or  both. 

A5-5.  AFOTEC  Regulations  (AFOTECR) 
and  Pamphlets  (AFOTECP): 

a.  AFOTECR  23-1,  Missions  and  Or¬ 
ganizational  Structures.  This  regulation 
reflects  the  approved  functional  responsibil¬ 
ities  and  organizational  structures  necessary 
to  accomplish  the  mission  of  AFOTEC  com¬ 
mand,  staff,  and  operating  elements. 

b.  AFOTECP  171-203,  Volume  II, 
Micro  Omnivore  Users  Pamphlet.  This 
pamphlet  describes  the  operating  procedures 
for  each  of  the  eight  primary  Micro  Omnivore 
functions:  Logon,  Menu  Handler,  Telecom¬ 
munications,  Data  Base  Update,  Analysis. 
Query,  Manual  Input,  and  Utilities. 

c.  AFOTECP  400-2,  Qualitative  Facil¬ 
ity  Evaluation.  This  pamphlet  provides 
procedures  for  conducting  qualitative  evalua¬ 
tions  of  facilities. 

d.  AFOTECP  800-1,  OT&E  Lessons 
Learned  Program.  This  regulation  imple¬ 
ments  AFR  800-13,  Air  Force  Feedback 
Policy.  It  establishes  AFOTEC  policy,  as¬ 
signs  responsibilities,  and  outlines  procedures 
for  submitting,  processing,  storing,  and  re¬ 
trieving  lessons  learned  reports. 

e.  AFOTECP  800-2,  Volume  1,  Man¬ 
agement  of  Software  Operational  Test 
and  Evaluation.  This  pamphlet  is  the  first 
volume  of  a  series  of  pamphlets  prepared  by 
the  Software  Evaluation  Division  at  HQ 
AFOTEC.  The  volumes  provide  the  software 
evaluation  manager  and  the  deputy  for 
software  evaluation  information  needed  for 
planning,  conducting,  and  reporting  on  OT&E 
of  mission  critical  software.  The  pamphlets 
in  the  series  are: 

(1)  AFOTECP  800-2,  Volume  1,  Man¬ 
agement  of  Software  OT&E. 

(2)  AFOTECP  800-2,  Volume  2,  Soft¬ 
ware  Support  Life  Cycle  Process  Evaluation 
Guide. 

(3)  AFOTECP  800-2,  Volume  3,  Soft¬ 
ware  Maintainability  Evaluation  Guide. 

(4)  AFOTECP  800-2,  Volume  4,  Soft¬ 
ware  Usability  Evaluator’s  Guide. 

(5)  AFOTECP  800-2,  Volume  5,  Soft 
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ware  Support  Resources  Evaluation  Guide. 

(6)  AFOTECP  800-2,  Volume  6,  Software 
Maturity  Evaluation  Guide. 

(7)  AFOTECP  800-2,  Volume  7,  Risk 
Assessment  for  Software  Supportability  Guide 
(Draft). 

A5-6.  Other  Command.  Regulations  and 
Pamphlets: 

a.  General.  Other  commands  often 
publish  supplements  to  the  preceding  regu¬ 
lations.  Though  too  numerous  to  list  here, 
AFOTEC  logistics  personnel  should  consult 
with  their  counterparts  when  needing  infor¬ 
mation  contained  in  command-specific  regula¬ 
tions  and  pamphlets.  Also,  the  LG  reference 
library  located  in  LG3  contains  a  copy  of 
commonly  used  guidance  documents  from 
other  commands.  The  following  documents 
are  key  reference  sources  for  the  information 
contained  in  this  pamphlet. 

b.  AFLCR  66-15,  Product  Perform¬ 
ance.  This  regulation  contains  policy  and 
requirements  for  data  system  maintenance 
and  procedures  on  using  and  analyzing 
deficiency  data  reported  on  Air  Force  systems 
and  equipment. 

c.  AFLCP/AFSCP  800-34,  Acquisition 
Logistics  Management.  This  pamphlet  is 
a  publication  of  AFLC/XRX  and  AFSC/SDD 
which  provides  a  comprehensive  source  of 
reference  information  for  acquisition  logistics 
matters  within  AFLC  and  AFSC.  It  is  also 
a  very  useful  source  of  information  for  all 
commands  involved  in  the  acquisition  logis¬ 
tics  process. 

A5-7.  Department  of  Defense  Documen¬ 
tation: 

a.  Justification  for  Major  System  New 
Starts.  Each  major  system  acquisition 
program  requires  a  JMSNS,  to  be  approved 
by  the  SECDEF.  DOD  components  shall 
prepare  a  JMSNS  to  document  major  defi¬ 
ciencies  in  their  ability  to  meet  mission 
requirements.  The  most  important  part  of 
the  JMSNS  is  the  evaluation  of  current  and 
planned  capabilities  in  relation  to  the  pro¬ 
jected  threat.  The  evaluation  can  be  based 
on  a  deficiency  in  the  existing  capability, 
such  as  excessive  manpower,  logistics  support 
requirements,  ownership  costs,  inadequate 
system  readiness,  or  mission  performance. 
In  addition,  key  boundary  conditions  for 
satisfying  the  need  are  identified,  such  as 
logistics,  safety,  and  manpower  considera¬ 
tions.  The  JMSNS  is  the  document  on  which 
the  Milestone  I  decision  is  based.  The  MNS 
states  the  user’s  needs,  and  the  JMSNS 
states  those  needs  in  relation  to  the  mission 
elements  in  a  manner  that  allows  a  Mile¬ 


stone  I  review  and  decision.  'Source:  DOD1 

5000.2. ) 

b.  System  Concept  Paper.  The  SCP 
is  required  documentation  of  Milestone  I.  It 
supports  the  JMSNS  and  states  the  direction 
needed  from  the  Secretary  of  Defense.  I: 
describes  the  concepts  explored  up  to  Mile¬ 
stone  I,  including  any  that  may  have  been 
rejected;  the  basis  for  narrowing  the  list  of 
concepts,  when  appropriate:  and  the  results 
of  the  system  and  threat  interactive  analysis. 
The  SCP  describes  the  recommended  alterna¬ 
tive  concepts  to  be  carried  forth  into  the 
demonstration  and  validation  (D&V)  phase. 
It  identifies  mission  requirements  that  sig¬ 
nificantly  impact  system  design  features  and 
support  concepts.  DODI  5000.2  contains  the 
specific  format  for  an  SCP.  The  annexes 
contain  system  performance  thresholds  and 
resource  requirements  for  the  acquisition 
program. 

c.  Test  and  Evaluation  Master  Plan 
(TEMP): 

(1)  The  TEMP  is  the  primary  docu¬ 
ment  used  in  the  OSD  review  and  decision 
process  to  assess  the  adequacy  of  the  planned 
testing  and  evaluation.  As  such,  the  TEMP 
must  be  of  sufficient  scope  and  content  to 
explain  the  entire  T&E  program.  The  DOD 
component  (in  the  Air  Force,  usually  AFSC 
shall  prepare  and  submit,  before  Milestone  I 
and  each  subsequent  milestone,  a  TEMP  for 
OSD  approval. 

(2)  Each  TEMP  submitted  to  OSD  must 
relate  the  T&E  effort  directly  to  technical 
risks,  operational  issues  and  concepts,  system 
performance,  availability,  reliability,  main¬ 
tainability,  logistics  requirements,  and  major 
decision  points. 

(3)  The  TEMP  should  include  the  key 
operational  and  technical  effectiveness  and 
suitability  characteristics,  but  it  is  not  lim¬ 
ited  to  the  characteristics  identified  in  the 
decision  milestone  documentation.  These 
characteristics  must  be  clearly  defined,  and 
the  program  milestones  at  which  the  thresh¬ 
olds  will  be  or  have  been  demonstrated  will 
be  indicated.  Prior  to  Milestone  II,  while 
tradeoffs  of  characteristics  are  underway,  it 
may  not  be  possible  to  establish  firm  goals 
or  thresholds.  In  this  case,  those  aspects  of 
performance  will  be  identified  which  are 
critical  to  the  ability  of  the  system  to  ac¬ 
complish  its  mission.  (Source:  DODD 

5000.3. ) 

d.  Secretary  of  Defense  Decision 
Memorandum  (SDDM).  The  SDDM  docu¬ 
ments  each  milestone  decision,  establishes 
program  goals  and  thresholds,  reaffirms 
established  needs  and  program  objectives, 
authorizes  exceptions  to  acquisition  policy 


A5-4 


AFOTECP  400-1  Attachment  5  15  May  1991 


(when  appropriate),  and  provides  the  direc¬ 
tion  and  guidance  to  OSD,  organization  of 
the  joint  chiefs  of  staff  (OJCS),  and  the  DOD 
component  for  the  next  phase  of  acquisition. 
Upon  approval  of  the  JMSNS  by  an  SDDM 
and  designation  of  a  system  as  major,  the 
DOD  component  must  take  necessary  pro¬ 
gramming  action  to  incorporate  required 
resources  into  the  PPBS.  (Source:  DODI 
5000.2.) 

e.  Decision  Coordinating  Paper  (DCP): 

(1)  The  DCP  provides  top-level  documen¬ 
tation  for  use  by  DAB  members  in  arriving 
at  a  recommendation  for  the  SECDEF  at 
Milestone  II  (and  Milestone  III  if  required). 
It  includes  a  program  description,  revalida¬ 
tion  of  the  mission  need,  goals  and  thresh¬ 
olds,  a  summary  of  the  DOD  component’s 
acquisition  strategy,  system  and  program 
alternatives,  and  issues  affecting  the  decision. 

(2)  DODI  5000.2  governs  the  form  and 
content  of  the  DCP.  Its  format  and  annexes 
are  the  same  as  for  the  SCP.  At  Milestone 
II,  emphasis  should  be  placed  on  goals  and 
requirements  related  to  system  readiness, 
associated  cost,  and  related  logistic  risks.  At 
Milestone  III,  logistic  resources  and  capability 
to  be  acquired  during  production  and  their 
relationship  to  system  readiness  and  cost 
objectives  should  be  highlighted.  Relevant 
changes  to  information  provided  in  previous 
program  documentation  should  be  addressed 
at  each  milestone.  (Source:  DODI  5000.2.) 

f.  Integrated  Program  Summary  (IPS). 
In  accordance  with  DODI  5000.2,  the  IPS 
provides  more  specific  information  than  the 
DCP  regarding  the  implementation  plan  of 
the  component  for  the  complete  acquisition 
cycle,  with  emphasis  on  the  phase  the  pro¬ 
gram  is  entering.  DODI  5000.2,  attachment 
2  to  enclosure  5,  titled:  Manpower,  pertains 
to  logistics.  This  attachment  includes: 

(1)  A  list  of  each  unit  type  that  will 
operate  the  system  and  primary  system 
elements,  including  unit  types  that  provide 
intermediate  maintenance  of  system  com¬ 
ponents.  Examples  of  unit  types  are  "Fighter 
Squadron”  and  "Munitions  Maintenance 
Squadron." 

(2)  For  each  unit  type,  the  manning 
required  to  satisfy  the  most  demanding 
mission  (normally  combat  employment  but 
may  be  precombat  readiness  for  certain  naval 
vessels  and  systems  on  alert). 

A5-8.  Operating  Command/HQ  USAF 
Documentation.  The  ability  to  perform 
OT&E  which  is  responsive  to  program  deci¬ 
sion  needs  depends  on  the  availability  of 
system-specific  information.  For  major  sys¬ 
tems,  a  series  of  required  documents  is 


defined  to  provide  this  information. 

a.  Statement  of  Operational  Need. 
The  operating  command  develops  the  MNS 
to  identify  an  operational  deficiency  and  state 
the  need  for  a  new  or  improved  capability  for 
USAF  forces.  Operational  needs  are  based 
on  short-term  and  long-term  capability  objec¬ 
tives  and  may  result  from  a  projected  defi¬ 
ciency  or  obsolescence  in  existing  capabil¬ 
ities,  a  technological  opportunity,  or  an 
opportunity  to  reduce  operating  and  support 
costs.  The  MNS  provides  the  basic  justifi¬ 
cation  to  initiate  program  acquisition  or 
modification  proposals.  If  the  evaluation  of 
the  formal  MNS  indicates  that  Secretary  of 
Defense  (SECDEF)  or  the  Secretary  of  the 
Air  Force  (SAF)  approval  is  necessary’,  HQ 
USAF  prepares  a  JMSNS  to  support  the 
need  and  submits  it  to  the  SECDEF  or  SAF. 
as  appropriate.  (Source:  AFR  57-1.) 

b.  Operational  Requirements  Docu¬ 
ment  (ORD).  The  operating  command 
submits  a  ORD  for  each  funded  program 
tested  in  the  Program  Management  Directive. 
The  ORD  is  the  requirements  and  planning 
document  prepared  to  address  operational 
and  support  needs.  It  amplifies  and  refines 
the  MNS.  A  major  part  of  the  ORD  is  the 
requirements  correlation  matrix  (RCM).  The 
RCM  is  a  multicolumn  spreadsheet  whose 
purpose  is  to  document  and  track  the  for¬ 
mulation  of  and  changes  to  user  require¬ 
ments  as  they  evolve  through  the  program 
acquisition  process.  (Source:  AFR  57-1.) 

c.  Program  Management  Directive 
(PMD).  The  PMD  provides  Air  Staff  direc¬ 
tion  for  acquisition  and  modification  pro¬ 
grams.  It  directs  the  actions  of  the  im¬ 
plementing,  using,  supporting,  and  other 
participating  commands.  It  is  a  living  docu¬ 
ment  that  is  prepared  when  the  program  is 
initiated  and  is  updated  throughout  the 
program.  The  PMD  contains  evaluation 
criteria  (goals  and  thresholds)  established  in 
the  SDDM.  (Source:  AFR  800-2.) 

A5-9.  Implementing/Supporting  Com- 
mand/MAJCOM  Documentation: 

a.  Program  Management  Plan  (PMP): 

(1)  The  PMP  is  the  baseline  manage¬ 
ment  document  used  for  implementing  and 
planning  an  acquisition  program.  It  shows 
the  schedule  of  events  and  necessary  re¬ 
sources.  The  program  office  prepares  the 
PMP,  and  unless  otherwise  directed  in  the 
PMD,  the  program  manager  approves  it.  The 
PMP  contains  only  that  information  the 
program  manager  deems  necessary. 

(2)  The  PMP  directs  participating 
organizations  by  identifying  and  defining 
their  participation  and  support  responsibil- 
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ities.  Where  OT&E  is  concerned,  however, 
the  PMP  can  only  reflect  the  IOT&E  program 
agreed  upon  by  the  responsible  OT&E  agency 
and  the  program  manager  and  the  FOT&E 
program  planned  by  the  OT&E  agency. 

(3)  The  PMP  contains  the  R&M  man¬ 
agement  plan.  (Source:  AFSCP  800-3,  A 
Guide  for  Program  Management.) 

b.  Acquisition  Strategy.  Acquisition 
strategy  is  the  conceptual  basis  of  the  overall 
plan  that  a  program  manager  follows  in 
program  execution.  It  reflects  the  manage¬ 
ment  concepts  that  will  be  used  to  direct  and 
control  all  elements  of  the  acquisition  and 
responds  to  specific  goals  and  objectives  of 
the  program.  It  ensures  that  the  system 
being  acquired  satisfies  the  approved  mission 
need.  The  acquisition  strategy  will  evolve 
through  an  iterative  process,  becoming  in¬ 
creasingly  definitive  in  describing  the  rela¬ 
tionships  of  management,  technical,  business, 
resource,  force  structure,  support,  test,  and 
other  aspects  of  the  program. 

c.  Integrated  Logistics  Support  Plan 
(ILSP).  The  program  manager  and  the 
deputy  program  manager  for  logistics  (DPML) 
or  integrated  logistics  support  manager 
(ILSM)  develops  and  uses  the  ILSP  to  man¬ 
age  the  ILS  process.  This  includes  the 
horizontal  integration  of  the  ILS  elements 
(i.e.,  with  each  other),  as  well  as  their  verti¬ 
cal  integration  into  the  various  aspects  of 
program  planning,  engineering,  designing, 
testing,  evaluating,  production,  and  operation. 
It  also  includes  the  integration  of  support 
elements  with  the  mission  elements  of  a 
system  throughout  its  life  cycle  and  is  up¬ 
dated  as  the  program  evolves.  The  ILSP  is 
a  part  of  the  PMP  and,  when  approved, 
becomes  directive  on  all  participating  agen¬ 
cies.  (Sources:  AFR  800-8  and  AFLCP/ 
AFSCP  800-34.) 

d.  Integrated  Support  Plan  (ISP). 
The  ISP  is  an  iterative  document  prepared 
by  a  contractor  for  the  acquiring  activity.  It 
describes  the  contractor’s  plan  for  managing 
the  contractual  ILS  program,  for  complying 
with  the  specific  contractual  ILS  require¬ 
ments,  and  for  planning  any  operational 
support  functions  assigned  to  the  contractor. 
(Sources:  AFR  800-8  and  AFLCP/AFSCP 
800-34.) 

e.  Logistics  Support  Analysis  (LSA). 
The  LSA  is  an  analytical  logistics  effort 
within  the  systems  engineering  process  to 
identify,  define,  analyze,  quantify,  and  proc¬ 
ess  logistics  support  requirements.  The 
logistics  support  analysis  record  (LSAR)  is 
the  source  of  validated,  integrated,  and 
design-related  logistics  data  for  an  acquisition 
program.  The  system  contractor  performs 


LSA  when  required  by  the  contract  statement 
of  work.  There  are  four  functions  of  the  LSA 
process: 

(1)  Identify  the  qualitative  and  quanti¬ 
tative  logistics  considerations. 

(2)  Influence  the  system  and  equip¬ 
ment  design  for  logistics  considerations. 

(3)  Communicate  requirements  and 
provide  an  integration  influence. 

(4)  Assess  the  achievement  of  logistics 
objectives. 

(Sources:  AFR  800-8,  AFLCP/AFSCP  800-34. 
and  MIL-STD  1388.) 

f.  Contract  Statement  of  Work 
(SOW): 

(1)  The  SOW  provides  an  excellent 
source  of  information  for  those  involved  in 
test  and  evaluation.  It  spells  out  what  the 
contractor  is  to  provide  in  quantitative  and 
qualitative  terms  and  also  specifies  what 
system/subsystem  testing  is  required  and  in 
what  kinds  of  environments.  Categories  of 
information  of  specific  interest  will  include 
reliability,  maintainability,  and  support- 
ability  programs;  goals;  thresholds;  demon¬ 
stration  plans;  growth  projection  methodolo¬ 
gies;  support  equipment;  work  breakdown 
structure;  reporting  requirements;  life  cycle 
cost  plans;  and  logistics  support  plans. 

(2)  The  results  expected  from  the 
contractor  will  provide  the  baseline  for  sys¬ 
tem  performance  expectations,  since  they  will 
define  parameters;  establish  a  set  of  expected 
suitability  parameter  values  under  specified 
environment  conditions;  and  map  out  the 
reliability  (or  reliability  growth)  required  to 
be  demonstrated  at  various  times  during 
system  development,  production,  and  deploy¬ 
ment.  All  of  these  values  will  be  based  on 
the  system  operational  and  maintenance 
concepts  approved  at  the  time  of  contract 
award.  These  values  will  normally  be  "in¬ 
herent"  values,  representing  the  best  perform¬ 
ance  a  system  may  be  expected  to  achieve. 
They  must  be  modified  to  reflect  the  oper¬ 
ating  concepts  and  environments  expected  in 
field  system  use.  (Sources:  DODDs  5000.39 
and  5000.40;  AFRs  800-8  and  800-18,  and 
AFLCP/AFSCP  800-34.) 

g.  Work  Breakdown  Structure  (WBS): 

(1)  One  of  the  most  useful  manage¬ 
ment  tools  for  program  managers  in  both 
DOD  and  industry  is  the  WBS.  Both  groups 
of  managers  need  total  program  visibility  and 
timely  data  on  program  progress  and  problem 
areas.  A  WBS  provides  the  framework  for 
the  required  management  visibility,  cost 
estimating,  and  data  reporting  in  a  manner 
directly  related  to  the  systems  engineering 
process. 

(2)  As  the  term  implies,  a  WBS  breaks 
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a  total  job  or  program  into  its  component 
elements.  These  elements  can  then  be  dis¬ 
played  to  show  their  relationship  to  each 
other  and  to  the  program  as  a  whole.  A 
program  WBS  reflects  the  systems  engi¬ 
neering  and  management  planning  processes 
during  development  and  production  of  a 
particular  system.  It  provides  a  schematic 
portrayal  of  the  products  (hardware,  software, 
services,  and  other  work  tasks)  that  com¬ 
pletely  define  the  program.  It  provides  a 
means  for  effective  management  planning 
and  implementation  by  providing  the  various 
functional  managers  of  a  program  (those  who 
are  involved  in  development,  production, 
finance,  procurement,  and  logistics)  with  a 
common  reference  framework  for  communi¬ 
cating  and  making  decisions.  AFR  800-17 
established  the  policy  for  developing  and 
applying  WBS  in  the  acquisition  of  defense 
materiel.  (Sources:  AFLCP/AFSCP  800-34.) 

h.  Reliability  and  Maintainability 
(R&M)  Management  Plan.  The  R&M 
management  plan  is  developed  in  accordance 
with  AFR  800-18  by  the  program  manager  in 
concert  with  other  activities  of  the  imple¬ 
menting,  supporting,  and  operating  com¬ 
mands  and  test  agencies  and  is  issued  prior 
to  release  of  request  for  proposal  (RFP). 
This  management  plan  should  be  part  of  the 
PMP.  Contents  will  include  R&M  program 
objectives,  responsibilities  of  various  Air 
Force  activities,  procedures  for  determining 
and  achieving  R&M  objectives,  data  require¬ 
ments/analysis  procedures,  flow  of  test  and 
contractor  R&M  data,  internal  Air  Force 
R&M  evaluation  and  reporting  procedures, 
and  prediction  of  R&M  and  logistics  perform¬ 
ance  values.  (Source:  AFR  800-18.) 

i.  Life  Cycle  Cost  Management  Pro¬ 
gram/Plans: 

(1)  AFR  800-11,  Life  Cycle  Cost  Man¬ 
agement  Program,  establishes  the  life  cycle 
cost  management  program.  The  objective  of 
the  program  is  to  ensure  that  the  Air  Force 
acquires  products  which  satisfy  operational 
needs  while  providing  the  lowest  feasible  life 
cycle  cost  Basic  policy  is  that  the  Air  Force 
will  consider  the  fidl  impact  of  life  cycle  costs 
in  making  decisions  associated  with  selection, 
design,  development,  acquisition,  modification, 
repair,  or  use  of  defense  materiel. 

(2)  Life  cycle  cost  is  the  total  cost  to  the 
government  of  an  item  or  system  over  its 
life.  It  includes  acquisition  costs  (research 
and  development,  test  and  evaluation,  and 
production  including  the  initial  investment 
for  a  product  support  capability)  and  recur¬ 
ring  operating  and  support  costs  or  "cost  of 
ownership”  (operations,  maintenance,  and 
other  support).  Disposal  costs  are  sometimes 


included  when  the  ultimate  salvage  of  equip¬ 
ment  has  some  residual  value  or  a  significant 
disposal  cost. 

(3)  The  program  manager  is  respon¬ 
sible  for  implementing  a  life  cycle  cost  man¬ 
agement  program.  Review  of  any  plans  and 
documentation  resulting  from  program  im¬ 
plementation  may  provide  insight  into  the 
"cost  driver"  areas  of  concern  which  might  be 
evaluated  during  OT&E.  (Sources:  AFLCP 
AFSCP  800-34.) 

j.  System  Maintenance  Concept.  AFR 
66-14  contains  guidance  for  the  preparation 
of  the  system  maintenance  concept.  The 
maintenance  concept  form s  the  basis  for  ail 
logistics  planning  and,  along  with  the  opera¬ 
tional  concept,  establishes  the  framework  for 
hardware  design.  As  the  design  becomes 
more  definitive,  a  series  of  logistics  analyses 
are  performed  to  substantiate  proposed 
design  changes  and  to  develop  a  maintenance 
plan  for  operational  application.  The  fully 
developed  maintenance  plan  is  validated 
during  DT&E  and  OT&E.  The  maintenance 
plan,  when  compatible  with  the  production 
configuration,  is  provided  to  the  supporting 
and  operating  commands.  (Sources:  AFRs 
57-1  and  66-14.) 

k.  Lessons  Learned  Files: 

(1)  ALD/ER  is  responsible  for  devel¬ 
oping  and  operating  a  "lessons  learned 
system  to  perform  the  vital  feedback  func¬ 
tion.  The  Logistics  Performance  Division 
(ALD/ERT)  manages  the  lessons  learned 
program  and  the  corporate  repository.  The 
repository  contains  information  (submitted  by 
any  organization)  relevant  to  a  deficiency  or 
improvement  of  a  technical  or  nontechnical 
nature  concerning  subsystems,  materials, 
processes,  or  proceuures  which  impact  on 
current  operational  systems  or  systems  being 
acquired. 

(2)  The  corporate  memory  or  lessons 
learned  file  provides  a  method  of  sharing 
lessons  learned  through  experience  in  acquisi¬ 
tion  and  deployment  of  existing  systems. 
The  intent  is  to  provide  a  focal  point  where 
program  managers  can  obtain  feedback  on 
the  results  of  previous  design  and  acquisi¬ 
tion  management  decisions  and  practices. 
The  AFLC  lessons  are  classified  into  two 
basic  categories,  technical  and  nontechnical 
(management). 

(a)  Technical  lessons  learned  pertain 
to  the  design  features  of  a  system/subsy¬ 
stem/equipment  which  influence  reliability, 
maintainability,  availability,  and  support 
cost.  This  includes  supporting  operational  or 
test  software. 

(b)  Nontechnical  lessons  learned  deal 
with  program  management  and  logistics 
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support  planning  influences  such  as  proce¬ 
dural  deficiencies/improvements,  time-phasing 
of  program  office/integrated  logistics  support 
office  (PO/1LSO)  actions,  quality  assurance, 
and  other  logistics  support  considerations. 

(3)  AFOTEC  test  directors  are  also  charg¬ 
ed  with  reporting  on  lessons  learned  during 
the  planning  and  execution  of  OT&E  pro¬ 
grams.  After  screening  and  substantiation, 
these  reports  are  placed  on  file  in  the  OT&E 
data  bank  maintained  by  AFOTEC  for  use  by 
all  MAJCOMs. 

(4)  Other  sources  of  lessons  learned  may 
be  found  in  system  program  offices,  operating 
commands,  and  other  services.  Examination 
of  the  files  may  aid  in  identifying  key  readi¬ 
ness  and  cost  drivers  for  investigation  during 
T&E.  (Sources:  AFR  55-43  and  AFLCP/ 
AFSCP  800-34.) 

A5-10.  Miliary/DOD  Handbooks  and 
Standards.  Several  military/DOD  handbooks 
and  standards  describe  the  availability, 
reliability,  and  maintainability  concepts, 
parameters  and  various  evaluation  tech¬ 
niques.  Descriptions  of  those  most  pertinent 
to  the  suitability  evaluation  follow: 

a.  MIL-HDBK  108,  Interim  Quality 
Control  and  Reliability  -  Sampling  Pro¬ 
cedures  and  Tables  for  Life  and  Relia¬ 
bility  Testing.  This  handbook  describes 
general  and  specific  procedures,  plus  applica¬ 
tions  of  sampling  plans  where  life  tests  are 
terminated  at  a  preassigned  number  of 
failures  or  a  preassigned  time.  It  also  de¬ 
scribes  sequential  life  test  sampling  plans 
and  provides  operating  characteristic  curves 
which  can  be  used  in  OT&E  effort  as  well  as 
research  and  development. 

b.  MIL-HDBK  189,  Reliability  Growth 
Management.  This  handbook  provides  the 
concepts  and  principles  of  reliability  growth, 
advantages  of  managing  reliability  growth, 
and  guidelines  and  procedures  to  be  used  in 
managing  reliability  growth.  It  also  contains 
descriptions  of  commonly  used  reliability 
growth  models. 

c.  MIL-HDBK  472,  Maintainability 
Prediction.  This  handbook  provides  tech¬ 
niques  to  predict  the  maintainability  of  a 
system  in  quantitative  terms.  It  is  intended 
for  application  early  in  the  acquisition  cycle 
to  highlight  areas  of  poor  maintainability 
design  for  product  improvement,  modification, 
or  a  change  in  design.  The  handbook  con¬ 
tains  a  series  of  prediction  procedures  for 
using  comparability  data  to  predict  the 
maintainability  of  new  systems. 

d.  DOD  3235. 1-H,  Test  and  Evaluation 
of  System  Reliability,  Availability,  and 
Maintainability — a  Primer.  This  handbook 


presents  concepts  and  techniques  for  design¬ 
ing  test  plans  to  verify  that  previously  estab¬ 
lished  system  suitability  requirements  have 
been  achieved.  It  provides  various  statistical 
concepts  and  techniques  required  to  thor¬ 
oughly  understand  the  relationships  among 
test  design,  assessment,  and  projection  of 
population  characteristics. 

e.  MIL- STD  105,  Sampling  Procedures 
and  Table  for  Inspection  by  Attributes. 
This  standard  provides  procedures  and  tables 
for  planning  and  conducting  inspections, 
whereby  either  the  unit  or  product  is  clas¬ 
sified  simply  as  defective  or  nondefective  or 
the  number  of  defects  in  the  unit  or  produce 
is  counted  with  respect  to  a  given  require¬ 
ment  or  set  of  requirements. 

f.  MIL-STD  470,  Maintainability  Pro¬ 
gram  Requirements.  This  standard  estab¬ 
lishes  the  requirements  for  a  maintainability 
program  and  provides  guidelines  for  the 
preparation  of  a  maintainability  plan  in  the 
development  of  a  system. 

g.  MIL-STD  471,  Maintainability  Dem¬ 
onstration.  This  standard  provides  proce¬ 
dures  and  test  methods  for  quantitative  and 
qualitative  maintainability  requirements.  It 
also  provides  for  qualitative  assessment  of 
other  elements  of  integrated  logistics  support 
such  as  technical  orders,  personnel,  support 
equipment,  provisioning,  and  maintenance 
concepts. 

h.  MIL-STD  490,  Specification  Prac¬ 
tices.  This  standard  covers  the  preparation 
of  specifications,  including  their  format  and 
content.  It  may  be  useful  to  understand  the 
application  of  contract  specifications  to  the 
system  being  evaluated. 

i.  MIL-STD  721,  Definitions  of  Effec¬ 
tiveness  Terms  for  Reliability,  Maintain¬ 
ability,  Human  Factors,  and  Safety.  This 
standard  provides  precise,  clear  definitions  to 
reduce  the  possibility  of  conflict,  duplication, 
or  incorrect  interpretation  of  the  meaning  of 
a  term.  It  is  an  authority  for  standardiza¬ 
tion,  not  an  all-inclusive  list  of  R&M  terms. 

j.  MIL-STD  756,  Reliability  Modeling 
and  Prediction.  This  standard  establishes 
uniform  procedures  for  predicting  the  quanti¬ 
tative  reliability  of  aircraft,  missiles,  satel¬ 
lites,  electronic  equipment,  and  subdivisions 
of  them  throughout  the  development  phases 
to  reveal  design  weaknesses  and  to  form  a 
basis  for  apportionment  of  reliability  require¬ 
ments  to  the  various  subdivisions  of  the 
product. 

k.  MIL-STD  781,  Reliability  Testing 
for  Engineering  Development,  Qualifica¬ 
tion,  and  Production.  This  standard 
covers  reliability  qualification  and  production 
acceptance  tests.  It  is  used  to  standardize 
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reliability  tests  for  more  direct  comparison  of 
test  results.  The  equipment  tests  outlined  in 
this  standard  do  not  replace  specified  func¬ 
tional  or  environmental  tests. 

l.  M3L-STD  785,  Reliability  Program 
for  Systems  and  Equipment  Develop¬ 
ment  and  Production.  This  standard 
establishes  uniform  criteria  for  a  reliability 
program  and  provides  guidelines  for  pre¬ 
paring  and  implementing  a  reliability  pro¬ 
gram  plan,  including  content  and  format. 

m.  MIL-STD  1388,  Logistics  Support 
Analysis.  This  standard  provides  task 
descriptions  for  the  performance  of  logistics 
support  analysis.  It  contains  an  overall  list 
of  logistics  support  analysis  tasks  from  which 
applicable  tasks  may  be  selected  and  tailored 
to  a  specific  analysis  program  and  includes 
the  rationale  for  selecting  and  tailoring  task 
descriptions. 

n.  MIL-STD  1543,  Reliability  Program 
Requirements  for  Space  and  Missile 
Systems.  This  standard  establishes  uniform 
reliability  program  practices  and  procedures 
for  use  during  design,  development,  fabrica¬ 
tion,  test,  and  operation  of  space  and  missile 
systems. 

o.  MIL-STD  1635  (EC),  Reliability 
Growth  Testing.  This  standard  covers  the 
requirements  and  procedures  for  reliability 
development  (growth)  tests  conducted  during 
the  hardware  development  phase.  These 
tests  provide  engineering  information  on  the 
failure  modes  and  mechanisms  of  a  test  item 
under  natural  and  induced  environmental 
conditions  of  military  operations.  Reliability 
improvements  (growth)  will  result  when  fail¬ 
ure  modes  and  mechanisms  are  identified 
and  their  recurrence  prevented  through 
implementation  of  corrective  action. 

p.  MIL-STD  2068(AS),  Reliability 
Development  Tests.  This  standard  estab¬ 
lishes  requirements  and  procedures  for  a 
reliability  development  test  to  implement  the 
MIL-STD  785  requirement  for  such  a  test. 


It  is  applicable  to  Navel  Air  Systems  Com¬ 
mand. 

q.  MIL-STD  2074  (AS),  Failure  Classi¬ 
fication  for  Reliability  Testing.  This 
standard  establishes  criteria  for  classification 
of  failures  occurring  during  reliability  test. 
Note:  Appendix  B  of  MIL-STD  2068  (AS; 
contains  equivalent  criteria  for  reliability 
growth  assessments. 

r.  DOD-STD  2167A,  Defense  System 
Software  Development.  This  standard 
establishes  uniform  requirements  for  software 
development  that  are  applicable  throughout 
the  system  life  cycle.  It  provides  a  basis  for 
government  insight  into  a  contractor's  soft¬ 
ware  development,  testing,  and  evaluation 
efforts.  In  addition,  it  provides  a  means  for 
establishing,  evaluating,  and  maintaining 
quality  in  software  and  associated  documen¬ 
tation. 

s.  DOD-STD  2168,  Defense  System 
Software  Quality  Program.  This  standard 
contains  requirements  for  the  development, 
documentation,  and  implementation  of  a 
software  quality  program  to  be  applied 
during  the  acquisition,  development,  and 
support  of  software  systems.  This  program 
includes  planning  for  and  conducting  evalua¬ 
tions  of  the  quality  of  software,  associated 
documentation,  and  related  activities  and 
planning  for  and  conducting  the  follow-up 
activities  necessary  to  ensure  timely  and 
effective  resolution  of  problems.  This  incor¬ 
porates  the  applicable  requirements  of  MIL- 
STD  1520  and  MIL-STD  1535. 

t.  DOD-STD  5200.28,  Department  of 
Defense  Trusted  Computer  System  Eval¬ 
uation  Criteria.  This  standard  defines 
evaluation  criteria  to  classify  systems  into 
four  broad  hierarchical  divisions  of  enhanced 
security  protection.  It  provides  a  basis  for 
the  evaluation  of  effectiveness  of  security 
controls  built  into  automatic  data  processing 
system  products. 
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DEFINITIONS 


This  attachment  provides  definitions  of  terms 
most  commonly  vised  during  OT&E.  As  a 
general  rule,  the  definitions  provided  are 
currently  in  use  in  the  Air  Force  and  have 
been  extracted  verbatim  from  other  directives 
(DODIs/DODDs,  AFRs,  MIL-STDs,  etc.).  A 
limited  number  of  terms  are  unique  to  a  test 
program  for  which  specific  definitions  were 
developed.  (See  chapter  13  for  diagnostics 
definitions.) 

Abort.  Failure  to  accomplish  a  mission  for 
any  reason  other  than  enemy  action.  It  may 
occur  at  any  point  from  initiation  of  opera¬ 
tion  to  destination.  (JCS  Pub  1) 

Acceptance  Tests.  Those  tests  performed 
to  demonstrate  that  a  specific  lot  of  articles 
has  been  manufactured  to  specifications. 
(AFR  80-14) 

Acquisition.  The  procurement  of  ownership 
of  real  property  by  any  means  exclusive  of 
lease  agreements.  The  process  consists  of 
planning,  designing,  producing,  and  distrib¬ 
uting  a  weapon  system/equipment.  Acquisi¬ 
tion  in  this  sense  includes  the  concept  ex¬ 
ploration,  validation,  FSD  production,  and 
deployment/operational  phases  of  the  weapon 
systems/equipment  projects.  (DOD  7040.2) 

Acquisition  Cost.  A  term  used  within  DOD 
to  denote  the  aggregation  of  costs  to  develop, 
produce,  and  deploy  a  weapon  system  in  its 
operational  environment.  It  commences  with 
the  conceptual  phase  and  is  completed  when 
the  last  production  unit  is  delivered  to  the 
using  command.  It  excludes  all  operational 
activities  associated  with  the  mission  applica¬ 
tion  of  the  acquired  weapon  system.  (AFSCR 
800-6) 

Acquisition  Process.  Normally,  this  con¬ 
sists  of  discrete  logical  phases  separated  by 
major  decision  points,  called  milestones 
(DODD  5000.1).  The  phases  span  the  life 
cycle  of  a  weapon  system  and  provide  a 
means  of  progessively  translating  broadly 
stated  mission  needs  into  well-defined  sys¬ 
tem-specific  requirements. 

a.  Phase  I,  Concept  Exploration  and 
Definition.  This  phase  begins  with  Mile¬ 
stone  O,  Concept  Studies  Approval.  Its  pur¬ 
pose  is  to  explore  various  material  alterna¬ 
tives  to  satisfying  the  documented  mission 
need,  define  the  most  promising  system 
concepts,  develop  supporting  analyses/ 
information  for  the  Milestone  I  decision,  and 


develop  a  proposed  acquisition  strategy  and 
initial  program  objectives  for  cost,  schedule, 
and  performance  (DODI  5000.2). 

b.  Phase  n,  Demonstration  and  Vali¬ 
dation.  This  phase  begins  with  Milestone  I, 
Concept  Demonstration  Approval.  Its  pur¬ 
pose  is  to  prove  technologies  and  processes 
critical  to  the  most  promising  system  con¬ 
cepts  are  understood  and  attainable,  better 
define  critical  design  characteristics,  develop 
analyses/information  to  support  a  Milestone 
II  derision,  and  establish  a  proposed  develop¬ 
ment  baseline  (DODI  5000.2). 

c.  Phase  m.  Engineering  and  Manu¬ 
facturing  Development.  This  phase  begins 
with  Milestone  II,  Development  Approval. 
Its  purpose  is  to  translate  the  preferred 
design  approach  into  a  stable  system  design, 
validate  the  manufacturing  or  production 
process,  and  demonstrate  through  testing, 
that  system  capabilities  meet  contract  specifi¬ 
cation  requirements  and  minimum  acceptable 
operational  performance  requirements  (DODI 
5000.2). 

d.  Phase  IV,  Production  and  Deploy¬ 
ment.  This  phase  begins  with  Milestone  III, 
Production  Approval.  Its  purpose  is  to 
establish  a  stable,  efficient  production  and 
support  base,  achieve  an  operational  capabil¬ 
ity  that  satisfies  the  identified  mission  needs, 
and  conduct  follow-on  operational  and  produc¬ 
tion  verification  testing  (DODI  5000.2). 

e.  Phase  V,  Operations  and  Support. 
This  phase  begins  with  Milestone  IV,  Major 
Modication  Approval  (as  required).  Its  pur¬ 
pose  is  to  ensure  fielded  system  continues  to 
provide  the  capabilities  required  to  meet  the 
identified  mission  need  and  identify  short¬ 
comings  or  deficiencies  that  must  be  cor¬ 
rected  (DODI  5000.2). 

Air  Force  System  Acquisition  Review 
Council  (AFSARC).  The  AFSARC  is  an 
advisory  council  established  by  and  func¬ 
tioning  for  the  Secretary  of  the  Air  Force 
(SAF)  which  provides  a  forum  for  the  review 
of  major  acquisition  programs  and  Air  Force 
designated  acquisition  programs  (AFDAP). 
Other  programs  may  receive  AFSARC  re¬ 
views  as  determined  by  the  SAF.  (AFR 
800-2) 

Air  Force  Weapon  System  Improvement 
Group  (AFWSIG).  Provides  the  DCS  for 
Logistics  and  Engineering  (HQ  USAF/LE) 
with  an  OT&E  suitability  assessment  in 
support  of  LE  inputs  to  the  AFSARC. 
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Assessment.  Provides  information  about  a 
system’s  capabilities  without  assigning 
ratings.  This  term  applies  when  user  re¬ 
quirements  are  not  available  or  are  not 
appropriate  for  the  phase  of  development. 
Information  is  needed  to  support  the  user  or 
decision-making  process.  Quantitative  or 
qualitative  test  criteria  will  generally  not  be 
available  to  support  these  objectives. 

Automatic  Test  Equipment  (ATE)  Pro¬ 
gram.  The  actions  taken  to  ascertain  to 
what  extent  depots  may  use  automatic  elec¬ 
tronic  test  equipment  in  promoting  the  effi¬ 
cient  and  economical  maintenance  and  analy¬ 
sis  of  weapon  system  equipment.  The  term 
also  encompasses  (1)  the  necessary  research 
and  development  actions,  (2)  control  of  ac¬ 
quisition  and  application,  and  (3)  controls  to 
ensure  that  adequate  consideration  is  given 
to  future  system  design  of  programmed 
weapon  systems  and/or  subsystems  for  ulti¬ 
mate  compatibility  with  programmed  or 
existing  depot  ATE.  (AFLCR  66-26) 

Availability.  A  measure  of  the  degree  to 
which  an  item  is  in  the  operable  and  com- 
mittable  state  when  the  mission  is  called  for 
at  any  random  point  in  time.  Availability  is 
dependent  on  reliability,  maintainability,  and 
logistics  supportability.  (AFR  800-18) 

a.  Availability  (Armament,  Munitions). 
The  percentage  of  assigned  munitions  capable 
of  performing  the  specified  missions  at  any 
random  point  in  time.  (AFR  800-18) 

b.  Availability  (Intercontinental  Ballis¬ 
tic  Missile  (ICBM)).  The  percentage  of  a 
missile  force  capable  of  commitment  to  the 
launch  sequence  at  any  random  point  in 
time.  (AFR  800-18) 

Before  Flight  Abort.  An  attempted  sortie 
that  fails  to  become  airborne  because  of  a 
failure.  The  criteria  used  will  be  those 
applied  by  the  predominant  using  command. 

Bench  Check-  A  workshop  check  for  the 
condition,  completeness,  or  working  order  of 
a  piece  of  equipment.  (AFM  67-1) 

Bench  Check  Serviceable  Rate.  The 
percentage  of  items  removed  from  an  end 
item  because  of  a  suspected  failure  for  which 
the  failure  was  not  confirmed  during  bench 
check  using  available  skills,  test  equipment, 
and  technical  data. 

Cannibalization.  The  authorized  removal 
of  specific  components  from  one  item  of  Air 
Force  property  for  installation  on  another 
item  of  Air  Force  property  to  meet  priority 


requirements  with  the  obligation  of  replacing 
the  removed  components.  ‘.AFM  67-1 

Cannibalization  Rate.  A  measure  of  on 
equipment  cannibalization  actions  ‘.removals 
performed  to  keep  an  end  item  in  operation¬ 
ally  ready  condition.  The  rate  may  be  ex¬ 
pressed  as  average  cannibalization  per  sortie, 
per  1,000  flying  hours,  or  other  life  units. 

Captive-Carry  Reliability.  The  probability 
that  a  weapon  will  remain  failure  free  while 
properly  loaded  on  the  host  aircraft.  When 
the  weapon  is  commanded  to  launch,  it  is 
expected  to  perform  a  successful  launch 
(AFR  800-18) 

Captive-Carry  Time.  The  cumulative  flying 
time  accrued  by  a  missile  or  other  ordnance 
end  item  while  attached  to  a  bomb  rack  or 
missile  launcher  on  an  aircraft  from  the 
beginning  of  the  first  flight  to  the  moment 
the  missile  or  ordnance  item  is  launched. 

Compatibility.  The  capability  of  two  or 
more  items/systems  to  exist  or  function  as 
elements  of  a  larger  system  or  environment 
without  mutual  interference.  (AFR  30-14.) 

Computer  Resources  Life  Cycle  Manage¬ 
ment  Plan  (CRLCMP).  This  plan  provides 
the  details  for  planning  and  implementing 
the  technical  and  managerial  responsibilities 
and  supporting  elements  required  to  ac¬ 
complish  software  development  and  life  cycle 
support.  MIL-STD  2167  requires  the  devel¬ 
opment  of  the  CRLCMP. 

Contractor  Logistics  Support.  The  collec¬ 
tion  of  logistics  support  activities  provided 
under  contract  to  a  using  command. 

Corrective  Maintenance.  All  actions 
performed  as  a  result  of  a  failure  to  restore 
an  item  to  a  specified  condition.  Corrective 
maintenance  can  include  any  or  all  of  the 
following  steps:  localization,  isolation,  disas¬ 
sembly,  interchange,  reassembly,  alignment, 
and  checkout.  (MIL-STD  72 1C) 

Critical  Design  Review.  A  formal  review 
conducted  on  each  configuration  item  before 
fabrication/production  design  release  to  en¬ 
sure  that  the  detail  design  solutions,  as 
reflected  in  the  draft  part  II  product  specifi¬ 
cation  and  engineering  drawings,  satisfy 
performance  requirements  established  by  the 
part  I  development  specification. 

Critical  Failure.  A  failure  or  combination 
of  failures  (hardware  or  software)  that  pre- 
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vent  an  item  from  performing  ;+s  specified 
missions. 

Critical  Issues.  Those  aspects  of  a  system’s 
capability,  either  operational,  technical,  or 
other,  that  must  be  questioned  before  a 
system’s  overall  worth  can  be  estimated,  and 
that  are  of  primary  importance  to  the  deci¬ 
sion  authority  in  reaching  a  decision  to  allow 
the  system  to  advance  into  the  next  acquisi¬ 
tion  phase.  (DODD  5000.3) 

Defence  Acquisition  Board  (DAB).  An 
advisory  council  established  by  and  func¬ 
tioning  for  the  Secretary  of  Defense 
(SECDEF)  to  apprise  the  SECDEF  of  the 
program  status  and  readiness  of  a  major 
defense  system  prior  to  proceeding  to  the 
next  phase  in  the  acquisition  process.  (DODI 
5000.2  and  AFR  800-2) 

Depot-Level  Maintenance.  Maintenance 
performed  on  materiel  requiring  major  over¬ 
haul  or  a  complete  rebuild  of  parts,  assem¬ 
blies,  subassemblies,  and  end  items,  including 
the  manufacture  of  parts,  modification,  test¬ 
ing,  and  reclamation  as  required.  Depot 
maintenance  supports  lower  levels  of  mainte¬ 
nance  by  (1)  providing  technical  assistance 
and  performing  that  maintenance  beyond 
their  responsibility  or  capability,  (2)  providing 
stocks  of  serviceable  equipment,  and  (3)  using 
more  extensive  facilities  for  repair  than  are 
available  in  organizational-  or  field-level 
maintenance  activities.  (AFM  67-1) 

Depot-Level  Maintenance  Support. 
Maintenance  and  modification  support  ac¬ 
complished  or  provided  by  Air  Force  Logistics 
Command  (AFLC).  It  includes  (1)  organiza¬ 
tional-  and  intermediate-level  maintenance  or 
modification  work  which  cannot  be  economi¬ 
cally  accomplished  within  the  using  com¬ 
mand’s  total  resources  and  is  so  certified  by 
the  using  command  headquarters,  and  (2) 
depot-level  maintenance  or  modification  work 
which,  because  of  the  complexity  of  the  job, 
requires  special  skills,  tools,  equipment,  or 
facilities  available  only  at  a  depot-level 
facility.  (AFR  66-1) 

Depot  Maintenance  Facility.  This  is  a 
military  or  contractor  facility  that  performs 
depot-level  maintenance  modification  of 
aircraft/missiles.  (AFR  66-3) 

Design  Deficiency.  Any  material  condition 
which  is  in  conformance  with  contractual 
requirements,  yet  limits  or  precludes  use  of 
material  in  the  intended  manner  and/or  for 
the  intended  purpose.  Those  deficiencies 


cannot  be  corrected  except  through  a  design 
change.  (DODD  7700.12) 

Design  to  Cost.  A  concept  that  directs 
action  during  the  design  phase  of  weapon 
system  to  establish  cost  as  a  key  parameter 
together  with  schedule  and  system  perform¬ 
ance  criteria.  System  design  and  develop¬ 
ment  are  continuously  evaluated  against  cost 
requirements  with  the  same  rigor  as  applied 
to  technical  requirements.  (AFR  173-1) 

Development  Test  and  Evaluation 
(DT&E).  DT&E  is  that  T&E  conducted  to 
assist  the  engineering  design  and  develop¬ 
ment  process  and  to  verify  attainment  of 
technical  performance  specifications  and 
objectives.  DT&E  is  normally  accomplished 
or  managed  by  the  DOD  component’s  svs- 
terns,  hardware/software  integration,  related 
software,  and  prototype  or  full-scale  engi¬ 
neering  development  models  of  the  system. 
T&E  of  compatibility  and  interoperability 
with  existing  or  planned  equipment  and 
systems  are  also  included.  (DODD  5000.3; 

Dormant  Reliability.  The  probability  that 
an  item  will  remain  failure  free  for  a  speci¬ 
fied  period  of  time  in  a  nonoperating  mode 
under  stated  environmental  conditions. 
When  the  system  is  removed  from  the  dor¬ 
mant  stage,  it  is  expected  to  perform  within 
specifications.  (AFR  800-18) 

Downtime  Per  Sortie.  For  a  specified 
period  of  time,  the  total  time  the  system  is 
not  mission  capable,  maintenance  (NMCM). 
scheduled  or  unscheduled;  not  mission  capa¬ 
ble,  supply  (NMCS);  or  not  mission  capable, 
both  I'NMCB),  scheduled  or  unscheduled;  in 
clock  hours  divided  by  the  number  of  sorties. 
(AFR  800-18) 

End  Item.  A  final  combination  of  end 
products,  component  parts,  and/or  materials 
ready  for  its  intended  use,  e.g.,  aircraft, 
ships,  tanks,  mobile  machine  ship.  (AFR 
400-3) 

Engineering  Change  Proposal  (ECP). 
The  document  for  proposing  any  design 
change  to  an  item,  facility,  part,  and  so 
forth,  delivered  or  to  be  delivered,  which  will 
require  revision  to  the  contract  specifications 
or  engineering  drawings,  or  the  documents 
referenced  that  are  approved  or  authorized 
for  applicable  items  under  government  con¬ 
tracts.  (AFR  400-3) 

Evaluation.  The  review  and  analysis  of 
qualitative  and/or  quantitative  data  obtained 
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from  design  review,  hardware  inspection, 
testing,  and/or  operational  usage  of  equip¬ 
ment.  (AFR  80-14) 

Exploratory  and  Advanced  Development. 
Exploratory  development  includes  all  efforts 
directed  to  solving  specific  military  problems 
short  of  major  development  projects.  Ad¬ 
vanced  development  includes  all  projects  that 
have  moveu  into  the  development  of  hard¬ 
ware  for  experimental  or  functional  test. 
'AFR  800-18) 

Failure  Rate.  The  total  number  of  failures 
within  an  item  population  divided  by  the 
total  number  of  life  units  expended  by  that 
population,  during  a  particular  measurement 
interval  under  stated  condition  .  (MIL-STD 
721C) 

Follow-on  Operational  Test  and  Evalua¬ 
tion  (FOT&E).  That  test  and  evaluation 
conducted  after  IOT&E  to  continue  and 
refine  the  estimates  made  during  the  IOT&E, 
to  evaluate  changes,  and  to  reevaluate  the 
system  to  ensure  that  it  continues  to  meet 
operational  needs  and  retain  its  effectiveness 
in  a  new  environment  or  against  a  new 
threat.  (AF  80-14) 

Foreign  Object  Damage  (FOD).  Damage 
to  or  malfunction  of  an  aircraft,  missile,  or 
drone  caused  by  an  object  that  is  alien  to  an 
area  or  system,  being  ingested  by,  or  lodged 
in  a  mechanism.  (AFR  66-33) 

Functional  Configuration  Audit  (FCA). 
The  formal  examination  of  functional  test 
data  for  a  configuration  item,  prior  to  accept¬ 
ance,  to  verify  that  the  item  has  achieved 
specified  performance.  (AFR  65-3) 

GoaL  Goals  are  levels  of  performance  (es¬ 
tablished  by  the  user)  above  that  required, 
which,  if  achieved,  will  provide  additional 
operational  capability.  Goals  are  not  normal¬ 
ly  addressed  by  HQ  AFOTEC  whose  primary 
concern  is  requirement.  (AFR  57-1) 

Government-Furnished  Equipment 
(GFE).  Items  in  the  possession  of  or  ac¬ 
quired  directly  by  the  government  and  de¬ 
livered  to  or  otherwise  made  available  to  the 
contractor.  (AFR  70-9) 

Government  Industry  Data  Exchange 
Program  (GIDEP).  An  Army-,  Navy-,  Air 
Force-,  and  NASA-sponsored  program  for  the 
exchange  of  data  among  government  agencies 
and  industry  to  reduce  the  costs  of  investiga¬ 
tive  efforts  on  parts  and  materials.  (AFR 


300-13) 

Ground  Abort.  Any  aircraft  confirmed  b;. 
maintenance  as  operational  and  ready  for 
flight  that  fails  to  launch  for  any  system 
malfunction/failure'reject. 

How-Malfunctioned  Code.  A  three-digit 
number  used  to  provide  a  description  of  the 
trouble  on  or  in  the  equipment  or  component 
(Appropriate  system  work  unit  code  (WUC 
manual.) 

Human  Engineering.  The  application  or 
knowledge  of  man’s  capabilities  and  limita¬ 
tions  to  the  planning,  design,  development, 
and  testing  of  aerospace  systems,  equipment, 
and  facilities  to  achieve  optimum  personnel 
safety,  comfort,  and  effectiveness  compatible 
with  systems  requirements.  (AFM  11-1) 

Human  Factors.  Those  factors  that  con¬ 
tribute  to  the  optimization  of  a  system  by 
integrating  the  human  performance  necessar. 
to  operate,  maintain,  support,  and  control  the 
system  in  its  intended  operational  environ¬ 
ment.  (AFR  80-14) 

Implementing  Command.  The  command 
responsible  for  managing  the  program  or 
development  and  acquisition  of  the  system, 
subsystem,  or  item  of  equipment.  (AFR 
800-18) 

In-Flight  Abort.  A  failure  of  an  airborne 
aircraft  so  that  it  cannot  effectively  accom¬ 
plish  its  primary  or  alternate  scheduled 
mission  because  of  a  reported  malfunction. 

Inherent  R&M  Value.  A  measure  of  relia¬ 
bility  or  maintainability  that  includes  only 
the  effects  of  an  item  design  and  its  applica¬ 
tion  and  assumes  an  ideal  operation  and 
support  environment.  (MIL-STD  7210 

Initial  Operational  Capability  (IOC).  The 
first  attainment  of  the  capability  to  employ 
effectively  a  weapon,  item  of  equipment,  or 
system  of  approved  specific  characteristics, 
and  which  is  manned  or  operated  by  an 
adequately  trained,  equipped,  and  supported 
military/unit  or  force.  (JCS  F*ub  1) 

Initial  Operational  Test  and  Evaluation 
(IOT&E).  OT&E  conducted  prior  to  the  first 
major  production  decision.  (AFR  80-14) 

Initial  Spares  Support  List  (ISSL).  A  list 
of  spares  and  repair  parts  and  quantities 
required  for  organizational  and  field  main¬ 
tenance  initial  support  of  an  end  item  for  a 
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given  period  of  time.  Quantities  established 
/for  ISSLr  will  be  equal  to  initial  base  stock- 
age  objective.  (AFR  400-3) 

In-Process  Review  (IPR).  A  review  of  a 
material  development  project  conducted  at 
critical  points  in  the  development  cycle  for 
the  purpose  of  evaluating  the  status  of  the 
project,  accomplishing  effective  coordination, 
and  facilitating  proper  and  timely  decisions 
bearing  on  the  future  course  of  the  project. 
(AFR  800-8) 

Integrated  Diagnostics.  The  process  of 
efficiently  utilizing  the  most  effective  combi¬ 
nation  of  a  system’s  automated,  semiauto- 
mated,  and  manual  diagnostics  resources  in 
a  total  system  approach  both  during  the 
mission,  and  subsequently,  at  each  successive 
maintenance  level. 

Integrated  Logistics  Support  (ILS).  A 
composite  of  all  support  necessary  to  ensure 
effective,  economical  support  of  a  system 
throughout  its  life  cycle.  (AFR  800-8) 

Intermediate-Level  Maintenance.  Main¬ 
tenance  that  is  normally  the  responsibility  of, 
and  performed  by,  designated  maintenance 
activities  for  direct  support  of  using  organiza¬ 
tions.  Its  phrases  normally  consist  of  cali¬ 
brating,  repairing,  or  replacing  damaged  or 
unserviceable  parts,  components,  or  assem¬ 
blies;  modification  of  material,  emergency 
manufacturing  of  unavailable  parts;  and 
providing  technical  assistance  to  using  or¬ 
ganizations.  Intermediate  maintenance  is 
normally  accomplished  by  the  using  com¬ 
mands  in  fixed  or  mobile  shops.  (AFR  66-1 
and  AFM  67-1) 

Interoperability.  The  ability  of  systems, 
units,  or  forces  to  provide  service  to,  and 
accept  services  from,  other  systems,  units,  or 
forces  for  their  mutual  effectiveness.  (AFR 
80-14) 

Latent  Defect.  A  flaw  or  other  imperfection 
in  an  article  discovered  after  delivery  to  the 
government.  Such  defects  are  inherent 
weaknesses  normally  not  detected  by  exami¬ 
nation  or  routine  test  but  present  at  time  of 
manufacture.  (AFM  67-1) 

Launch  Availability.  The  probability  that 
a  launch  vehicle  system  will  be  ready  for 
launch  in  any  preselected  launch  window. 
(AFR  800-18) 

Launch  and  Flight  Reliability  (LFR). 
The  probability  of  a  missile  system  that  is 


available  for  commitment  to  the  launch 
sequence,  responding  to  a  valid  launch  com¬ 
mand,  and  successfully  completing  the  launch 
and  flight  with  detonation  of  a  given  war¬ 
head  within  accuracv  requirements.  (AFR 
800-18) 

Level  of  Repair  Analysis.  A  term  as¬ 
signed  to  a  technique  which  establishes  (1) 
whether  an  item  should  be  repaired;  (2)  at 
what  maintenance  level,  i.e.,  organizational, 
intermediate,  or  depot;  or  (3)  if  the  item 
should  be  discarded.  (AFP  800-7) 

Levels  of  Protection.  The  degree  of  pres¬ 
ervation,  packaging,  and  packing  required  to 
prevent  deterioration  or  damage  to  supplies 
and  equipment  because  of  the  hazards  to 
which  they  may  be  subjected  during  ship¬ 
ment  and  storage.  (MIL-STD  1290 

Life  Cycle  Cost.  The  total  cost  of  an  item 
or  system  over  its  full  life.  It  includes  the 
cost  of  development,  production,  ownership 
(operation,  maintenance,  support,  etc.),  and 
where  applicable,  disposal.  (AFR  800-11) 

Line-Replaceable  Unit  (LRU): 

a.  An  item  that  is  normally  removed  and 
replaced  as  a  single  unit  to  correct  a  defi¬ 
ciency  or  malfunction  on  a  weapon  or  support 
system  and  item  of  equipment.  Such  items 
have  a  distinctive  stock  number  for  which 
repairs  may  be  locally  authorized  to  support 
the  removal  and  replacement  action.  These 
items  are  repair  cycle  assets  subject  to/due 
in  from  maintenance  (DIFM)  controls  (TO 
00-20-3)  and  may  be  disassembled  into  sepa¬ 
rate  components  during  shop  processing. 

b.  The  components,  shop-replaceable  units 
(SRU),  may  also  be  repair  cycle  assets  sub¬ 
ject  to  DIFM  controls  if  they  are  processed 
separately  and  spares  are  locally  authorized 
and  maintained  to  support  intermediate-level 
repair  of  the  LRU.  (AFM  400-1) 

Logistics: 

a.  The  science  of  planning  and  carrying 
out  the  movement  and  maintenance  of  forces. 
In  its  most  comprehensive  sense,  those  as¬ 
pects  of  military  operations  that  deal  with 
design  and  development,  acquisition,  storage, 
movement,  distribution,  maintenance,  evacua¬ 
tion,  and  disposition  of  material;  movement, 
evacuation,  and  hospitalization  of  personnel; 
acquisition  or  construction,  maintenance, 
operation,  and  disposition  of  facilities;  and 
acquisition  or  furnishing  of  services.  (JCS 
Pub  1  and  AFP  800-7) 

b.  The  functional  fields  of  military  opera¬ 
tions  concerned  with  materiel  requirements; 
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production  planning  and  scheduling;  acquisi¬ 
tion,  inventory  management,  storage,  mainte¬ 
nance,  distribution  and  disposal  of  materiel, 
supplies,  tools,  and  equipment;  transporta¬ 
tion,  telecommunications,  petroleum,  and 
other  logistical  services;  supply  cataloging, 
standardization,  and  quality  control;  commer¬ 
cial  and  industrial  activities  and  facilities 
including  industrial  equipment;  and  vulner¬ 
ability  of  resources  to  attack  damage. 
(DODD  5000.8) 

c.  The  phase  of  military  operations  involv¬ 
ing  procurement,  delivery,  storage,  shipment, 
and  scheduling  of  military  supplies,  including 
personnel.  (AFLCR  72-2) 

d.  The  determination  of  initial  and  follow- 
on  requirements  and  the  procurement,  stor¬ 
age,  transportation,  distribution,  maintenance, 
quality  control,  and  disposal  of  material  and 
related  services  for  the  military  forces. 
(AFLCR  400-15) 

Logistics  Assessment.  An  evaluation  of  the 
logistics  support  required  to  support  par¬ 
ticular  military  operations  in  a  theater  of 
operations,  country,  or  area. 

Logistics  Assessment  Review  (LAR). 
Conducted  by  HQ  USAF/LE  before  key  ac¬ 
quisition  milestones,  i.e.,  AFSARC  reviews, 
OSD  program  reviews,  and  DAB  reviews. 
HQ  USAF/LE  reviews  logistics  data  to  ensure 
that  logistics  areas  are  given  adequate  con¬ 
sideration.  The  developing,  supporting,  and 
operating  commands  provide  information  to 
the  assessment.  AFOTEC  provides  the  oper¬ 
ational  suitability  assessment.  (AFOTECR 
500-3) 

Logistics  Concept.  A  plan  of  how  to  build 
up  or  support  a  military  force,  i.e.,  to  provide 
supplies,  equipment,  transportation,  main¬ 
tenance,  etc.  (AFM  67-1) 

Logistics  Planning.  The  determination  of 
the  logistics  posture  to  be  established  for 
support  of  a  weapon/support  system  program 
based  on  prescribed  mission  objectives  to  be 
achieved.  (AFM  11-1,  AFP  800-7) 

Logistics  Process.  A  task  or  group  of 
interrelated  logistics  tasks  designed  to  pro¬ 
duce  a  desired  result  independent  of  the 
organizational  arrangement  employed.  (AFM 
400-2) 

Logistics  Resources.  The  support  person¬ 
nel  and  material  required  by  an  item  to 
ensure  its  mission  performance.  It  includes 
such  things  as  tools,  test  equipment,  repair 
parts,  facilities,  technical  manual,  and  ad¬ 


ministrative  supply  procedures  necessary  u 
ensure  the  availability  of  these  resources 
when  needed.  (AFP  S00-7) 

Logistics  Support  Analysis  (LSA).  A 
process  by  which  the  logistics  support  neces¬ 
sary  for  a  new  system/equipment  is  iden¬ 
tified.  It  includes  the  determination  and 
establishment  of  logistics  support  design 
constraints,  consideration  of  those  constraints 
in  the  design  of  the  'hardware''  portion  of 
the  system,  and  analysis  of  design  to  validate 
the  logistics  support  feasibility  of  the  design 
and  identify  and  document  the  logistics 
support  resources  which  must  be  provided,  as 
a  part  of  the  system/equipment,  to  the  oper¬ 
ating  forces.  Analytical  techniques  used  to 
determine  limited  aspects  of  logistics  support 
requirements  are  a  part  of  the  overall  LSA 
process.  (An  example  would  be  operational 
sequential  diagramming  used  to  determine 
operator  tasks,  task  times,  and  skills.)  '.AFP 
800-7  and  MIL-STD  1388) 

Logistics  Support  Costs.  Costs  associated 
with  supporting  an  item,  to  include  (when 
obtainable)  costs  of  base  labor,  base  materiel, 
costs  to  replace  condemnations,  transportation 
and  shipping  costs  for  nonbase  reparable 
items,  technology  repair  center  costs,  and 
others  when  the  cost  is  quantifiable  and 
meaningful  for  effective  analysis.  (AFLCR 
400-16) 

Logistics  Support  Elements.  Principal 
logistics  elements  that  must  be  properly 
integrated  to  achieve  economical  and  effective 
support  of  a  system  or  equipment  throughout 
its  life  cycle.  The  elements  are  R&M  inter¬ 
face;  maintenance  planning;  support  equip¬ 
ment;  facilities;  training,  technical  data; 
personnel;  supply  support;  packaging,  han¬ 
dling,  transportation,  and  storage;  logistics 
support  resource  funds;  logistics  support 
management  information;  design  interface; 
computer  resources  support;  energy  manage¬ 
ment;  survivability;  and  ILS  T&E.  (AFR 
800-8) 

Logistics  Supportability.  How  well  the 
composite  of  support  considerations  necessary 
to  achieve  the  effective  and  economical  sup¬ 
port  of  a  system  or  equipment  for  its  life 
cycle  meets  stated  quantitative  and  qualita¬ 
tive  requirements.  This  includes  integrated 
logistics  support  and  logistics-related  O&S 
cost  considerations.  (AFR  80-14) 

Logistics  System.  A  group  of  related  and 
sequential  actions  or  documents  required 
and/or  used  to  accomplish  one  or  more  ele- 
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merits  of  the  AFLC  mission  (AFR  23-2)  or  to 
provide  support  in  the  accomplishment  of  the 
mission.  This  includes  operational  and 
management  functions.  The  term  "logistics 
system"  encompasses  such  terms  as  data 
system,  automated  data  system,  management 
system,  business  system,  and  similar  terms. 
In  general,  any  group  of  actions  for  processes 
performed  repetitively  as  differentiated  from 
nonrecurring  special  projects  is  considered  a 
logistics  system  or  subsystem.  (AFLCM 
400-4) 

Low-Rate  Initial  Production  (LRIP).  The 
production  of  a  system  in  limited  quantity  to 
be  used  in  OT&E  for  verification  of  produc¬ 
tion  engineering  and  design  maturity  and  to 
establish  a  production  base. 

Maintainability.  The  measure  of  the  ability 
of  an  item  to  be  retained  in  or  restored  to 
specified  condition  when  maintenance  is 
performed  by  personnel  having  specified  skill 
levels,  using  prescribed  procedures  and 
resources,  at  each  prescribed  level  of  main¬ 
tenance  and  repair.  Also  reference  MIL-STD 
721  for  the  general  definition  of  maintainabil¬ 
ity.  (AFR  800-18) 

Maintenance.  All  actions  necessary  for 
retaining  material  in  or  restoring  it  to  a 
serviceable  condition.  Maintenance  includes 
servicing,  repair,  modification,  modernization, 
overhauls,  inspection,  condition  determina¬ 
tion,  corrosion  control,  and  initial  provision¬ 
ing  of  support  items.  (AFR  66-14) 

Maintenance  Downtime  Per  Sortie.  For 
a  specified  period  of  time,  the  total  time  the 
system  is  NMCM  and  NMCB,  scheduled,  in 
clock  hours,  divided  by  the  number  of  sorties. 
(AFR  800-18) 

Maintenance  Engineering.  The  developing 
of  maintenance  concept,  criteria,  and  techni¬ 
cal  requirements--during  the  conceptual  and 
definition  phases-to  be  applied  and  main¬ 
tained  in  the  operational  phase,  to  ensure 
timely,  adequate,  and  economic  maintenance 
support  of  systems  and  equipment.  (AFR 
66-1) 

Maintenance  Man-Hours  Per  Life  Unit 
(MMH/LU).  The  base-level,  direct  main¬ 
tenance  man-hours  required  to  support  a 
system  divided  by  the  number  of  life  units 
(e.g.,  MMH/S,  MMH/FH).  This  includes 
direct  maintenance  man-hours  identified  by 
the  Standard  Reporting  Designator  of  the 
weapon  system  and  its  installed  or  removed 
equipment.  (AFR  800-18) 


Maintenance  Personnel  Per  Operational 
Unit.  (User  of  this  term  needs  to  define  thv 
operational  unit.;  The  number  of  main¬ 
tenance  personnel  that  will  be  required  :c 
support  an  operational  unit  (excluding  depot- 
level  and  other  manpower  that  is  excluded 
from  maintenance  planning  factors  by  AFR 
26-3)  under  specified  operating  and  main¬ 
tenance  concept.  (AFR  800-18) 

Mature  System.  An  operational  system  is 
considered  mature  when  its  R&M  characteris¬ 
tics  cease  to  improve  significantly  with  con¬ 
tinued  use.  System,  subsystems,  and  com¬ 
ponents  all  mature  at  various  rates  for 
varying  lengths  of  time.  Unless  otherwise 
specified,  a  system  will  be  considered  to  have 
mature  R&M  characteristics  2  years  after  the 
initial  operational  capability  date.  (AFR 
800-18) 

Mean  Downtime  (MDT).  The  average 
elapsed  time  between  loss  of  mission-capable 
status  and  restoration  of  the  system  to 
mission-capable  status.  (AFR  800-18) 

Mean  Man-Hours  To  Repair.  The  totai 
corrective  base-level  man-hours  divided  by 
the  total  of  equipment  corrective  maintenance 
events  for  a  given  period  of  time.  (AFR 
800-18) 

Mean  Mission  Duration  (MMD).  The 
average  interval  of  time  over  which  a  space 
system  is  expected  to  operate  without  mission 
failure.  (AFR  800-18) 

Mean  Time  Between  Critical  Failures 
(MTBCF).  The  average  time  between  failure 
of  essential  system  functions.  (AFR  800-1S. 

Mean  Time  Between  Demands  (MTBD). 
A  measure  of  the  system  reliability  param¬ 
eter  related  to  demand  for  logistics  support: 
the  total  number  of  system  life  units  (e.g.. 
flying  hours,  sorties,  etc.)  divided  by  the  total 
number  of  item  demands  on  the  supply 
system  during  a  stated  period  of  time. 
Demands  are  defined  in  AJFLCR  57-4,  Re¬ 
coverable  Consumption  Item  Requirements 
System.  (AFR  800-18  and  MIL-STD  7210 

Mean  Time  Between  Downing  Events. 
A  measure  of  the  system  reliability  param¬ 
eter  related  to  availability  and  readiness. 
The  total  number  of  system  life  units  divided 
by  the  total  number  of  events  in  which  the 
system  becomes  unavailable  to  initiate  its 
mission(s)  during  a  stated  period  of  time. 
(MIL-STD  7210 
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Mean  Time  Between  Failures  (MTBF). 
(A  contract  term  only.)  A  basic  measure  of 
reliability  for  repairable  items.  The  mean 
number  of  life  units  during  which  all  parts 
of  the  item  perform  within  their  specified 
limits  during  a  particular  measurement 
interval  under  stated  conditions.  (MIL-STD 
7210 

Mean  Time  Between  Maintenance 
(MTBM).  The  total  life  units  (for  example, 
operating  hours,  flight  hours,  rounds)  divided 
by  the  total  number  of  maintenance  (base- 
level)  events  for  a  specific  period  of  time. 
(AFR  800-18) 

a.  Mean  Time  Between  Maintenance 
(Induced).  The  average  time  between  the 
on-equipment  corrective  events  associated 
with  malfunctions  resulting  from  other  than 
internal  design  and  manufacturing  character¬ 
istics,  for  example,  improper  maintenance, 
operator  error,  foreign  object  damage,  and 
failures  caused  by  malfunction  of  associated 
equipment.  (AFR  800-18) 

b.  Mean  Time  Between  Maintenance 
(Inherent).  The  average  time  between  the 
on-equipment  corrective  events  associated 
with  malfunctions  resulting  from  internal 
design  and  manufacturing  characteristics. 
(AFR  800-18) 

c.  Mean  Time  Between  Maintenance 
(No  Defect).  The  average  time  between  the 
on-equipment  corrective  events  associated 
with  equipment  which  have  no  confirmed 
malfunction,  such  as  removals  which  subse¬ 
quently  bench  check  satisfactory.  (AFR 
800-18) 

d.  Mean  Time  Between  Maintenance 
(Preventive).  The  average  time  between 
maintenance  events  including  removals, 
replacement,  or  reinstallation  associated  with 
scheduled  maintenance  or  time  changes. 
(AFR  800-18) 

Mean  Time  Between  Removal  (MTBR). 
A  measure  of  the  system  reliability  param¬ 
eter  related  to  demand  for  logistics  support. 
The  total  number  of  system  life  units  divided 
by  the  total  number  of  items  removed  from 
that  system  during  a  stated  period  of  time. 
This  term  is  defined  to  exclude  removals 
performed  to  facilitate  other  maintenance  and 
removals  for  TCTOs  (product  improvement). 
(AFR  800-18,  MIL-STD  721C) 

Mean  Time  To  Failure  (MTTF).  A  basic 
measure  of  reliability  for  nonrepayable  items. 
The  total  number  of  life  units  of  an  item 
divided  by  the  total  number  of  failures 
within  that  population  during  a  particular 
measurement  interval  under  stated  condi¬ 


tions.  (MIL-STD  7210 

Mean  Time  To  Repair  (MTTR).  <A  con¬ 
tract  term  only.)  A  basic  measure  of  main¬ 
tainability:  the  sum  of  corrective  mainte¬ 
nance  times  at  any  specific  level  of  repair 
divided  by  the  total  number  of  failures 
within  an  item  repaired  at  that  level  during 
a  particular  interval  under  stated  conditions 
(MIL-STD  721C) 

Mean  Time  To  Restore  System  (MTTRS). 
A  measure  of  the  system  maintainability 
parameter  related  to  availability  and  readi¬ 
ness.  The  total  corrective  maintenance  time 
associated  with  downing  events  divided  by 
the  total  number  of  downing  events  during 
a  stated  period  of  time.  (Excludes  time  for 
off-system  maintenance  and  repair  of  de¬ 
tached  components.)  (MIL-STD  72 1C) 

Mean  Time  To  Service  (MTTS).  A  meas¬ 
ure  of  an  on-system  maintainability  charac¬ 
teristic  related  to  servicing  that  is  calculated 
by  dividing  the  total  scheduled  crew/opera- 
tor/driver  servicing  time  by  the  number  of 
times  the  item  was  serviced.  (MIL-STD 
721C) 

Mission  Reliability.  A  measure  of  the 
ability  of  a  system  to  complete  its  planned 
mission  or  function.  (AFR  800-18) 

Mission  Time  To  Restore  Functions 
(MTTRF).  A  measure  of  mission  maintaina¬ 
bility:  the  total  corrective  critical  failure 
maintenance  time,  divided  by  the  total  num¬ 
ber  of  critical  failures,  during  the  course  of 
a  specified  mission  profile.  (MIL-STD  72 1C 

Network  Report  Level  Analysis  (NRLA). 
NRLA  is  the  preferred  means  of  performing 
RLA  It  solves  the  RLA  problem  for  LRUs 
and  SRUs.  It  solves  the  problems  of  failure 
mode,  recognizing  that  an  LRU  may  fail  in 
any  of  several  different  ways.  It  treats  its 
individual  failure  modes  as  part  of  the  over¬ 
all  problem.  It  treats  the  problem  of  shared 
SE  successfully. 

Not  Mission  Capable  (NMC).  A  status 
code  meaning  that  the  system  or  equipment 
cannot  perform  any  of  its  primary  missions. 
It  can  be  followed  by  a  reason  code  meaning 
maintenance  (M),  supply  (S),  or  both  (B). 
(AFR  66-14) 

Not  Reparable  This  Station  (NRTS).  A 
status  condition  determined  during  shop 
processing  of  an  item  used  to  indicate  that 
the  item  cannot  be  repaired  at  base  level 
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because  of  lack  of  authorization,  technical 
skills,  parts,  facilities,  manpower,  or  any 
other  causes.  (TO  00-20-1) 

Off-Equipment  Maintenance.  In-shop 
maintenance  actions  performed  on  removed 
components,  except  complete  aircraft  engines. 
(AFR  800-18) 

OMNIVORE/MICRO-OMNIVORE.  A  data 
retrieval  and  analysis  system.  (AFOTECR 
171-202) 

On-Condition  Maintenance.  Application 
of  inspection  and  testing  procedures  and 
techniques  without  removal  or  disassembly 
that  allows  the  condition  of  the  equipment  to 
dictate  the  need  for  maintenance  or  the 
extent  of  repair/overhaul  required  to  restore 
serviceability.  (AFR  66-38) 

On-Equipment  Maintenance.  Maintenance 
actions  accomplished  on  a  complete  end  item 
such  as  aircraft,  trainers,  support  equipment, 
CEM  equipment,  complete  round  munitions, 
and  uninstalled  aircraft  engines.  (AFR 
800-18) 

On-Launcher  Reliability.  The  percentage 
of  ready  missiles  which  will  successfully 
complete  the  countdown  and  leave  the 
launcher  within  the  required  time  limits. 
(DODD  3100.1) 

Operating  Command.  The  command  or 
agency  primarily  responsible  for  the  opera¬ 
tional  employment  of  a  system,  subsystem,  or 
item  of  equipment;  it  is  also  a  participating 
command.  (AFR  800-18) 

Operational  Assessment  (OA).  The  opera¬ 
tional  test  agency’s  independent  appraisal  of 
the  status  of  operational  requirements  and 
IOT&E  readiness  and  the  progress,  from  an 
operational  perspective,  of  a  system’s  develop¬ 
ment. 

Operational  Effectiveness.  The  overall 
degree  of  mission  accomplishment  of  a  sys¬ 
tem  used  by  representative  personnel  in  the 
context  of  the  organization,  doctrine,  tactics, 
threat  (including  countermeasures  and  nu¬ 
clear  threats),  and  environment  in  the 
planned  or  operational  employment  of  the 
system.  (DODD  5000.3) 

Operational  Reliability  and  Maintainabil¬ 
ity.  A  measure  of  reliability,  maintainabil¬ 
ity,  or  availability  expressed  in  operational 
terms  that  includes  the  combined  effects  of 
item  design,  quality,  installation,  environ¬ 


ment,  operation,  maintenance,  repair,  fund¬ 
ing,  and  management  policy.  (AFR  S00-1S 

Operational  Suitability.  The  degree  to 
which  a  system  can  be  satisfactorily  placed 
in  field  use  with  consideration  being  given 
to  availability,  compatibility,  transportability, 
interoperability,  reliability,  wartime  usage 
rates,  maintainability,  safety,  human  factors, 
manpower  supportability,  logistic  support- 
ability,  documentation,  and  training  require¬ 
ments.  (DODI  5000.2) 

Operational  Test  and  Evaluation 
(OT&E).  OT&E  is  conducted  in  as  realistic 
conditions  as  possible  throughout  the  system 
life  cycle.  It  is  done  to  estimate  (or  to  refine 
estimates  of)  a  system’s  operational  effective¬ 
ness  and  operational  suitability,  to  identify 
any  operational  deficiencies,  and  to  identify 
the  need  for  any  modifications.  (AFR  80- 14; 

Operational  Utility  Evaluations  (OUE). 
OUEs  are  conducted  (1)  to  validate  a  concept 
and  (2)  to  expand  the  mission  of  an  existing 
(perhaps  modified)  weapon  system  to  a  sub¬ 
stantially  different  role  or  mission.  OUEs 
pertain  to  those  operational  tests  clearly  out¬ 
side  the  scope  of  existing  tests  (i.e.,  DT&Es. 
IOT&Es,  QOT&Es,  and  FOT&Es).  An  OUE 
may  be  conducted  before  Milestone  I  at 
which  time  the  focus  is  on  providing  informa¬ 
tion  necessary  to  support  or  validate  concept 
selection  (for  example,  type  of  system  to  best 
fill  the  operational  reconnaissance  needs  of 
theater  commanders  such  as  aircraft  vis-a- 
vis  remotely  piloted  vehicle  vis-a-vis  satellite 
systems).  Conversely,  OUEs  may  also  inves¬ 
tigate  the  feasibility  of  expanding  the  opera¬ 
tional  mission  of  an  existing/modified  system 
to  a  new  mission  or  scenario  (for  example, 
modifying  an  air-to-ground  antiarmor  missile 
so  it  can  be  employed  in  the  airfield  attack 
scenario  against  reverted  aircraft).  OUEs 
will  be  HQ  USAF  directed  and  AFOTEC  or 
MAJCOM  conducted  and  will  be  specifically 
limited  in  time  and  scope.  OUEs  will  nor¬ 
mally  be  accomplished  with  RDT&E  (3600) 
funds;  however,  in  some  instances,  OUEs 
may  be  more  appropriately  funded  through 
the  MAJCOM  account.  If  there  is  insuffi¬ 
cient  time  for  normal  funding  in  the  PPBS 
schedule,  funding  will  be  provided  by  HQ 
USAF  concurrently  with  the  direction  to 
conduct  an  OUE. 

Optimum  Repair  Level  Analysis  (ORLA). 
(A  subset  of  repair  level  analysis  (RLA)j 
(See  AFSCR/AFLCR  800-28,  Repair  Level 
Analysis  Program.)  A  trade  study  conducted 
by  a  contractor  as  part  of  the  system/equip- 
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ment  engineering  analysis  process.  ORLA 
provides  contractors  and  prospective  contrac¬ 
tors  with  a  basis  on  which  to  evolve  an 
"optimum"  approach  to  repair  recommenda¬ 
tions  concurrent  with  the  design  and  develop¬ 
ment  process.  (AFLCM/AFSCM  800-4  and 
AFLCR  66-26) 

Orbital  Availability.  The  percentage  of  on- 
orbit  time  that  a  space  system  is  capable  of 
performing  the  specified  missions.  (AFR 
800-18) 

Organizational-Level  Maintenance.  Main¬ 
tenance  that  is  the  responsibility  of  and 
performed  by  a  using  organization  on  its 
assigned  equipment.  Organizational  main¬ 
tenance  normally  consists  of  inspecting, 
servicing,  lubricating,  adjusting,  and  replac¬ 
ing  parts,  minor  assemblies,  and  subassem¬ 
blies.  (AFM  400-1) 

Partially  Mission  Capable  (PMC).  The 
percentage  of  possessed  time  that  a  system 
is  capable  of  performing  at  least  one  but  not 
all  of  its  assigned  wartime  missions.  PMC 
may  be  subdivided  into  partially  mission 
capable,  maintenance  (PMCM);  partially 
mission  capable,  supply  (PMCS);  and  partial¬ 
ly  mission  capable,  both  (PMCB).  (AFR 
66-14) 

Participating  Command.  A  command  or 
agency  designated  by  HQ  USAF  to  support 
and  advise  the  program  manager  during  the 
execution  of  a  program.  (AFR  800-14) 

Pilot  Production.  A  limited  production  run 
of  a  new  system  which  has  completed  en¬ 
gineering  development  and  for  which  the 
capability  to  mass  produce  the  item  for 
inventory  needs  to  be  demonstrated.  (AFR 
80-14) 

Possessed  Hours.  The  total  hours  in  a 
given  period  that  assigned  equipment  is 
under  the  operational  control  of  the  desig¬ 
nated  responsible  organization.  (AFR  800-18) 

Program  Element  Monitor  (PEM).  The 
individual  in  the  Air  Staff  designated  to 
exercise  overall  monitorship  over  a  program 
element.  (AFM  11-1) 

Prototype.  First  full-scale  functional  form 
of  a  new  system  on  which  the  design  of 
subsequent  production  items  is  patterned. 
(AFR  80-14) 

Qualification  Tests.  Those  tests  that  verify 
the  design  and  manufacturing  process  and 


thus  provide  a  baseline  for  subsequent  ac¬ 
ceptance  tests.  (AFR  SO- 14) 

Quick  Turnaround  Time.  The  clock  hours 
required  to  prepare  an  aircraft  for  immediate 
relaunch  after  termination  of  a  sortie.  (AFR 
800-18) 

Reliability.  The  probability  that  an  item 
will  perform  its  intended  function  for  a 
specified  interval  under  stated  conditions. 

a.  Logistics  Reliability.  A  measure  of 
a  system’s  capability  to  operate  as  planned 
under  the  defined  operational  and  support 
concepts  using  specified  logistics  resources 
(for  example,  spares  or  manpower).  Logistics 
reliability  recognizes  the  effect  of  all  occur¬ 
rences  that  place  a  demand  on  the  logistics 
support  system  even  when  mission  capability 
is  unaffected.  (AFR  800-18) 

b.  Mission  Reliability.  A  measure  of 
the  ability  of  a  system  to  complete  its 
planned  mission  or  function.  (AFR  800-18; 

Reliability  Analysis  Center  (RAC).  An 
official  DOD  contractor-operated  center  locat¬ 
ed  at  Rome  .Air  Development  Center  (RADC 
(code  RBRAC)  authorized  to  collect,  analyze, 
and  disseminate  reliability  data  and  informa¬ 
tion  on  microcircuits,  solid  state  devices, 
nonelectronic  parts  and  equipment,  and 
systems.  (AFR  800-18) 

Reliability-Centered  Maintenance  Pro¬ 
gram  (RCMP).  A  failure  modes  and  effects 
analysis  technique  for  significant  aircraft  and 
engine  structures,  assemblies,  and  items.  It 
used  a  decision  logic  procedure  based  on  the 
Arlines/Manufacturers’  Maintenance  Planning 
Document,  MSG-2.  This  structured  approach 
to  maintenance  requirements  analysis  iden¬ 
tifies  minimum  essential  requirements  con¬ 
sistent  with  safety  and  readiness.  (AFR 
66-14) 

R&M  Engineering.  That  set  of  design, 
development,  and  manufacturing  tasks  by 
which  R&M  requirements  are  achieved. 
(AFR  800-18) 

Repair-Level  Analysis  (RLA).  The  basic 
decisions  about  (1)  repair  versus  throwaway 
and  (2)  the  most  desirable  repair  posture. 
(AFSCR/AFLCR  800-28) 

Satellite  Design  Life.  The  expected  suc¬ 
cessful  orbital  operating  time  before  failure 
because  of  depletion  of  expendables.  (AFR 
800-18) 

Secretarial  Program  Review  (SPR).  The 
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SPR  is  conducted  to  advise  the  Secretary  of 
the  Air  Force  and  selected  service  staff  on 
the  research,  development,  and  production 
aspects  of  major  programs.  It  provides  an 
in-depth  evaluation  and  means  for  making 
decisions  on  all  aspects  of  selected  major 
systems.  (AFOTECR  500-3) 

Service  Report  (SR).  A  report  to  supply 
AFR  80-14  test  and  evaluation  data  used  in 
test  reports  on  operational  and  logistics 
supportability  of  new  systems  or  equipment 
prior  to  production  decision  and  during 
follow-on  testing.  (TO  00-35D-54) 

Shop-Replaceable  Unit  (SRU).  A  module 
for  an  LRU  which  can  be  removed  from  the 
LRU  at  an  intermediate  repair  facility. 
(AFLCP  57-13) 

Software  Error.  Any  human  action  or 
omission  in  the  software  that  results  in  the 
software  containing  a  fault. 

Software  Failure.  The  inability  of  a  sys¬ 
tem  or  system  component  to  perform  a  re¬ 
quired  function  within  specified  limits.  A 
failure  may  result  when  a  fault  is  encounter¬ 
ed. 

Software  Fault.  An  accidental  condition 
that  causes  a  functional  unit  to  fail  to  per¬ 
form  its  intended  function. 

Software  Maintainability.  A  measure  of 
che  effort  required  to  locate  and  correct  an 
error  in  the  software,  provide  enhancements 
to  existing  software,  or  add  new  software  to 
accomplish  additional  requirements  when 
these  maintenance  activities  are  performed 
by  personnel  having  specific  skill  levels, 
using  prescribed  procedures  and  resources. 

Software  Maturity.  A  measure  of  the 
software’s  progress  in  its  evolution  toward 
satisfaction  of  all  documented  user  require¬ 
ments. 

Software  Reliability.  Software  reliability 
is  the  probability  that  a  software  fault  will 
not  be  encountered  which  could  possibly 
cause  a  failure  during  a  specified  exposure 
period. 

Software  Supportability.  A  measure  of 
the  ability  to  modify  deployed  software  in 
support  of  both  static  and  dynamic  mission 
requirements.  Software  supportability  is  a 
composite  of  (1)  life  cycle  processes,  (2) 
support  resources,  and  (3)  software  main¬ 
tainability. 


Sortie  Generation  Rate.  The  number  o: 
sorties  that  can  be  flown  per  aircraft  per  da;, 
under  specified  operational  and  maintenance- 
concepts.  (AFR  800-13) 

Support  Equipment  (SE).  SE  includes  aY. 
equipment  required  to  perform  the  support 
function  except  that  which  is  an  integral  par: 
of  the  mission  equipment.  It  does  not  in¬ 
clude  any  of  the  equipment  required  to 
perform  mission  operations  functions.  SE 
should  be  interpreted  as  including  tools,  test 
equipment,  automatic  test  equipment  :  ATE 
(when  the  ATE  is  accomplishing  a  support 
function);  organizational,  intermediate,  ana 
technical  repair  center  SE;  and  related  com¬ 
puter  programs  and  software.  (AFLCR  65-2 

Supporting  Command.  The  command 
assigned  responsibility  for  providing  logistic- 
support.  It  assumes  program  management 
responsibility  from  the  implementing  com¬ 
mand;  it  may  also  be  a  participating  com¬ 
mand.  (AFR  800-2) 

System  Program  Office  (SPO).  The 
organization  comprised  of  technical  and 
business  management  and  administrative 
personnel  assigned  full  time  to  a  system 
program  director.  The  office  may  be  aug¬ 
mented  with  additional  personnel  from  par¬ 
ticipating  organizations.  (AFM  11-1) 

Technical  Order  (TO).  An  Air  Force 
publication  that  gives  specific  technical 
directives  and  information  with  respect  to  the 
inspection,  storage,  operation,  modification, 
and  maintenance  of  given  Air  Force  items 
and  equipment.  (AFR  8-2) 

Test  and  Evaluation  (T&E).  The  term 
"test"  denotes  any  project  or  program  de¬ 
signed  to  obtain,  verify,  and  provide  data  for 
the  evaluation  of  research  and  development 
other  than  laboratory  experiments;  progress 
in  accomplishing  development  objectives: 
performance  and  operational  capability  ct 
systems,  subsystems,  and  components;  and 
equipment  items.  The  term  "evaluation 
denotes  the  review  and  analysis  of  quantita¬ 
tive  data  produced  during  current  or  previous 
testing,  data  obtained  from  test  conducted  by 
other  government  agencies  and  contractors, 
from  operation  and  commercial  experience, 
or  combinations  thereof.  (AFR  80-14  and 
AFM  11-1) 

Test  Planning  Working  Group  (TPWG). 
When  specified  by  AFSC  program  direction, 
this  group  is  established  by  the  program 
manager  to  provide  a  forum  for  test-related 
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subjects;  to  assist  in  establishing  test  objec¬ 
tives  and  evaluation  baselines;  to  define 
organization,  responsibilities,  and  relation¬ 
ships;  to  estimate  costs  and  schedules;  and 
to  identify  needed  test  resources.  The  TPWG 
normally  includes  representatives  from  the 
SPO,  AFSC  test  agencies,  contractor,  AFO- 
TEC,  and  using  and  supporting  MAJCOMs. 
(AFSC  Sup  1  to  AFR  80-14) 

Test  Support  Group  (TSG).  Consists  of 
representatives  of  HQ  AFOTEC  directorates/ 
detachments  and  staff  offices  with  particular 
expertise  required  for  a  specific  AFOTEC- 
conducted/monitored  OT&E.  The  TSG  is 
chaired  and  directed  by  the  AFOTEC  test 
manager  and  provides  the  assistance  needed 
to  budget  for,  plan,  execute,  evaluate,  and 
report  on  a  specific  test  program. 

Threshold.  Quantitative  and  qualitative 
minimum  essential  levels  of  performance  or 
capability  to  permit  mission  accomplishment. 
These  levels  are  based  on  (1)  operational  and 
maintenance  concepts;  (2)  the  threat  esti¬ 
mate;  (3)  operationally  significant  perform¬ 
ance  levels  contained  in  documents  such  as 
the  PDM,  PMD,  or  TEMP;  and  (4)  the  capa¬ 
bilities  of  existing  systems  (when  valid  com¬ 
parison  can  be  made).  (AFR  55-43) 

Time  Compliance  Technical  Order 
(TCTO).  Directives  issued  to  provide  in¬ 
struction  to  Air  Force  activities  for  accom¬ 
plishing  "one-time"  changes,  modifications, 
or  inspections  of  equipment  or  installation  of 
new  equipment.  (AFLCR  171-91) 

Transportability.  The  capability  of  mate¬ 
rial  to  be  moved  by  towing,  self-propulsion, 
or  carrier  via  any  means,  such  as  railways, 
highways,  waterways,  pipelines,  oceans,  and 
airways.  (JCS  Pub  1) 

Unscheduled  Maintenance.  Unpredicted 
maintenance  that  requires  prompt  attention 
to  restore  equipment  serviceability.  (AFSCR 
66-7) 

Uptime  Ratio.  The  percent  of  possessed 
time  that  communications,  electronics,  and 
meteorological  (CEM)  systems  are  operation¬ 
al.  (AFM  65-662) 

Validation.  The  process  by  which  the  con¬ 
tractor  tests  TOs  for  technical  accuracy  and 
adequacy.  This  is  accomplished  by  testing 


the  maintenance  and  operating  instruction; 
on  the  equipment/ systems  for  which  the  TO 
was  written.  Validation  is  conducted  at  the 
contractor  facility  or  at  the  operational  site. 
(AFR  66-7) 

War  Readiness  Spares  Kit  (WRSK).  An 
air  transportable  package  of  spares  and 
repair  parts  required  to  sustain  planned 
wartime  or  contingency  operations  of  a  weap¬ 
on  system  for  a  specified  period  of  time 
pending  resupply.  WRSKs  will  include 
spares  and  repair  parts  for  aircraft,  vehicles, 
and  other  equipment,  as  appropriate. 
WRSKs  are  normally  pre-positioned  with  the 
using  unit.  (AFM  11-1) 

Wartime  Usage  Rates.  Rates  at  which 
systems  and  their  supporting  subsystems, 
support  equipment,  and  spares  are  con¬ 
sumed/used  under  war  conditions. 

Weapon  System  Reliability  (WSR).  The 
probability  that  a  system  will  complete  a 
specified  mission,  given  that  the  system  was 
initially  capable  of  performing  that  mission. 
WSR  is  a  measure  of  system  reliability  as  it 
affects  the  mission  but  excludes  factors  such 
as  probability  of  kill,  circular  error  probable, 
and  other  measures  of  capability.  (AFR 
800-18) 

Work  Breakdown  Structure  (WBS).  A 
product-oriented  family  tree  division  of  hard¬ 
ware,  software,  services,  and  other  work 
tasks  which  organizes,  defines,  and  graph¬ 
ically  displays  the  product  to  be  produced  as 
well  as  the  work  to  be  accomplished  to 
achieve  the  specified  product.  (AFSCP 
AFLCP  173-5,  DODD  7000.2) 

Work-Unit  Code  (WUC).  This  code  is  a 
five-position  code  used  to  identify  equipment 
being  worked  on  or  maintenance  actions. 
WUCs  which  have  a  zero  as  the  first  digit 
are  titled  support  general  codes  and  will  be 
found  in  all  applicable  -06  WUC  manuals. 
Support  general  codes  are  used  to  identify 
maintenance  actions  such  as  aircraft  ground 
handling,  look  phase  of  scheduled  inspections 
ground  safety,  etc.  WUCs  used  to  identify 
items  of  the  system  (e.g.,  components,  subsys¬ 
tems,  etc.)  may  have  as  the  first  digit  an 
alpha  or  numeric  designator  (other  than  zero) 
and  are  divided  into  broad  categories.  (AFM 
65-110) 
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ABBREVIATIONS 


The  following  abbreviations  common  to  logistics  assessments  are  currently  accepted  anc  in 
use  in  the  Air  Force.  Abbreviations  in  common  use  are  included  even  though  they  may  nc: 
be  used  in  this  pamphlet. 


ACMS 

A&CO 

AD 

ADP 

ADPE 

ADS 

AFCC 

AFDAP 

AFEWC 

AFFTC 

AFLC 

AFMSMT 

AFOLDS 

AFOTEC 

AFPG 

AFR 

AFSARC 

AFSC 


AFWSIG 

AGE 

V 

ALC 

ALD 

ALDT 

AMTS 

ANOVA 

A&M 

ARMS 

ASD 

ATC 


ATE 

AVISURS 

AVPR 

AWP 


achieved  availability 
aircraft 

advanced  configuration  management  system 
assembly  and  checkout 

Armament  Division  (Air  Force  Systems  Command) 

automatic  data  processing 

automatic  data  processing  equipment 

automated  data  system 

Air  Force  Communications  Command 

Air  Force  Designated  Acquisition  Programs 

Air  Force  Electronic  Warfare  Center 

Air  Force  Flight  Test  Center 

Air  Force  Logistics  Command 

Air  Force  maintenance  and  supply  management  team 

Air  Force  on-line  data  systems 

Air  Force  Operational  Test  and  Evaluation  Center 

Air  Force  planning  guide 

Air  Force  regulation 

Air  Force  Systems  Acquisition  Review  Council 

Air  Force  System  Command 

Air  Force  specialty  code 

Air  Force  Weapon  Support  Improvement  Group 

aerospace  ground  equipment 

inherent  availability 

Air  Logistics  Center 

Acquisition  Logistics  Division 

administrative  and  logistical  delay  time 

active  man-hour  task  summary 

analysis  of  variance 

operational  availability 

availability,  reliability,  and  maintainability 

ammunition  reporting  management  system 

Aeronautical  Systems  Division  (Air  Force  Systems  Command) 

Air  Training  Command 
action  taken  code 
automatic  test  equipment 

aerospace  vehicle  and  equipment  inventory  status  and  utilization  reporting 
system 

air  vehicle  performance  report 
awaiting  parts 


BFA  before  flight  aborts 

BITE  false  alarm 
BIT  built-in  test 

BITE  built-in  test  equipment 

BUS  base-level  inquiry  system 

BMD  Ballistic  Missile  Division  (equivalent  to  Space  Division) 


A7-2 


AFOTECP  400-1  Attachment  7  15  May  1991 


ABBREVIATIONS  (continued) 


c 

delay  time  in  corrective  maintenance 

CAP 

combat  air  patrol 

CAR 

command  assessment  review 

CBR 

chemical,  biological,  and  radiological 

CCB 

configuration  control  board 

CCR 

captive-carry  reliability 

CDEP 

common  data  extraction  program 

CDR 

critical  design  review 

CDRL 

contractor  data  requirements  list 

CEI 

configuration  end  item 
contractor  end  item 

CEM 

communication-electronic-meteorology 

CERT 

combined  environments  reliability  test 

CFD 

critical  faults  detected 

CFE 

contractor-furnished  equipment 

CMT 

corrective  maintenance  time 

CND 

cannot  duplicate 

COMO 

combat-oriented  maintenance  organization 

COMSEC 

communication  security 

CRISP 

computer  resources  integrated  support  plan 

CRWG 

computer  resources  working  group 

CSAF 

Chief  of  Staff,  United  States  Air  Force 

CSAS 

configuration  status  accounting  system 

CSRL 

common  strategic  rotary  launcher 

CTAT 

combat  turnaround  time 

CTDCS 

common  test  data  collection  system 

DAB 

Defense  Acquisition  Board 

DAD 

data  description  language 

DAE 

defense  acquisition  executive 

DAR 

Defense  Acquisition  Regulation  (formerly  ASPR) 

DART 

deficiency  analysis  review  technique 

DBA 

data  base  administration 

DBMS 

data  base  management  system 

DCM 

deputy  commander  for  maintenance 

D  CP 

decision  coordinating  paper 

DDC 

Defense  Documentation  Center 

DDT&E 

director,  development  test  and  evaluation 

DEW 

distant  early  warning 

DID 

data  item  description 

DIOS 

defense  integrated  data  system 

DIFM 

due  in  from  maintenance 

DLE 

deputy  for  logistics  evaluation 

DtSlE 

Defense  Logistics  Studies  Information  Exchange 

DMPA 

deployment  maintenance  plan  assessment 

DOD 

Department  of  Defense 

DPI 

data  processing  installation 

DPM 

development  program  manuals 

DPML 

deputy  program  manager  for  logistics 

DR 

dormant  reliability 

DSE 

deputy  for  software  evaluation 

DT 

downtime 

DT&E 

development  test  and  evaluation 
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DTIC 

DUEL 

D&V 

ECC 

ECMS 

ECP 

ED8 

EDSC 

El 

EMI 

EOC 

EOD 

ESC 

ESD 

ESS 

EW 

ETI 

FA 

FAD 

FAR 

FCA 

FCF 

FD 

FH 

FI 

FIIN 

FIT 

FMC 

FMEA 

FMECA 

FOD 

FOL 

FOT&E 

FRACAS 

FRDB 

FSC 

FSD 

FT 

FTD 

GFE 

GIDEP 

GSE 

HRL 

IAW 

ICBM 

IFA 

ILS 

ILSMT 


ABBREVIATIONS  (continued) 


Defense  Technical  Information  Center 
data  update  edit  language 
demonstration  and  validation 

extended  captive  carry 
engine  configuration  management  system 
engineering  change  proposal 
engineering  data  bank 
Engineering  Data  Service  Center 
end  item 

electromagnetic  interference 
equipment  operating  cycle 
explosive  ordnance  disposal 
Electronic  Security  Command 

Electronic  Systems  Division  (Air  Force  Systems  Command) 
environmental  stress  screening 
electronic  warfare 
elapsed  time  indicator 

false  alarm 

force/activity  designator 

Federal  Acquisition  Regulation  (formerly  DAR) 

functional  configuration  audit 

functional  check  flight 

fault  detection 

flight  hours 

fault  isolation 

federal  item  identification  number 

fault-isolation  test 

fully  mission  capable 

failure  modes  and  effects  analysis 

failure  modes,  effects,  and  criticality  analysis 

foreign  object  damage 

forward  operating  location 

follow-on  operational  test  and  evaluation 

failure  reporting  and  corrective  action  system 

failure  rate  data  bank 

federal  supply  classification 

full-scale  development 

free  time 

Foreign  Technology  Division 

government-furnished  equipment 
government-industry  data  exchange  program 
ground  support  equipment 

human  resource  laboratory 

in  accordance  with 

intercontinental  ballistic  missile 

in-flight  aborts 

integrated  logistics  support 

integrated  logistics  support  management  team 
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ILSP 

integrated  logistics  support  plan 

IM 

item  manager 

IMF 

integrated  maintenance  facility 

I/O 

input/output 

IOC 

initial  operational  capability 

IOT&E 

initial  operational  test  and  evaluation 

IPR 

in-process  review 

IPS 

integrated  program  summary 

IROS 

increase  reliability  of  operational  system 

ISD 

instructional  systems  development 

ISP 

integrated  support  plan 

ISSL 

initial  spares  support  list 

JMSNS 

justification  for  major  systems  new  start 

JRMET 

joint  reliability  and  maintainability  evaluation  team 

LA 

launch  availability 

LAR 

logistics  assessment  review 

LCC 

life  cycle  cost 

LOOM 

logistics  composite  model 

LFR 

launch  and  flight  reliability 

LG 

Directorate  of  Logistics,  AFOTEC 

LGI 

space  surveillance  and  injunction  systems  division 

LGM 

aircraft  systems  division 

LGOI 

logistics  operating  instruction 

LGW 

weapons  and  IC8M  systems  division 

LG4 

logistics  studies  and  analysis  division 

LG5 

software  analysis  division 

LOAP 

list  of  applicable  publications 

LRB 

logistics  review  board 

LRU 

line-replaceable  unit 

LSA 

logistics  support  analysis 

LSAR 

logistics  support  analysis  record 

M 

maintainability 

MAC 

Military  Airlift  Command 

MA/FH 

maintenance  actions  per  flying  hour 

MAJCOM 

major  command 

MC 

mission  capable 

MCOTEA 

Marine  Corps  Operational  Test  and  Evaluation  Agency 

MCS 

maintenance  cost  system 

MCT 

mean  corrective  time 

MO 

mission  duration 

MDC 

maintenance  data  collection 

MDCS 

maintenance  data  collection  system 

MDR 

maintenance  data  reporting 

MDT 

mean  downtime 

MEA 

maintenance  engineering  analysis 

MEAR 

maintenance  engineering  analysis  record 

MEP 

(Air  Force)  management  engineering  program 

MESL 

mission  essential  subsystem  list 

MFHBF 

mean  flying  hours  before  failure 

MHA 

man-hour  accounting 
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MHR 

MILAP 

MIL-HDBK 

MIL-SPEC 

MIL-STD 

MIP 

MIPR 

Mmax  C 

MMD 

MMDT 

MMH/CCH 

MMH/FH 

MMH/LU 

MMH/M 

MMH/MA 

MMH/OH 

MMH/S 

MMICS 

MMR 

MNS 

MOA 

MOB 

MOE 

MOOL 

MOT&E 

MR 

MRA&L 

MRF 

MRRT 

MRT 

MSBM 

MSK 

MSRT 

MST&E 

MTA 

MTBCF 

MTBD 

MTBDE 

MTBF 

MTBM 

MTBOS 

MTBR 

MTBUMA 

MTRRM 

MTS 

MTSCO 

MTTD 

MTTF 

MTTR 

MTTRF 

MTTRS 

MTTS 


mission  hardware  reliability 

maintenance  information  logically  analyzed  and  presented 

military  handbook 

military  specification 

military  standard 

material  improvement  program 

military  interdepartmental  purchase  request 

maximum  corrective  maintenance  time 

mean  mission  duration 

mean  maintenance  downtime 

maintenance  man-hours  per  captive-carry  hour 

maintenance  man-hours  per  flying  hour 

maintenance  man-hours  per  life  unit 

maintenance  man-hours  per  mission 

maintenance  man-hours  per  maintenance  action 

maintenance  man-hours  per  operating  hour 

maintenance  man-hours  per  sortie 

maintenance  management  information  and  control  system 

mean  man-hours  to  repair 

mission  need  statement 

memorandum  of  agreement 

main  operating  base 

measure  of  effectiveness 

mean  on-orbit  lifetime 

multinational  operational  test  and  evaluation 
material  reporting 

manpower  reserve  affairs  and  logistics 
milestone  reference  file 
mean  requisition  response  time 
mean  repair  time 

mean  sorties  between  maintenance 

mission  support  kit 

mean  supply  response  time 

multiservice  test  and  evaluation 

mean  time  to  assemble 

mean  time  between  critical  failures 

mean  time  between  demands 

mean  time  between  downing  events 

mean  time  between  failures 

mean  time  between  maintenance  actions 

mean  time  to  break  out  of  storage 

mean  time  between  removals 

mean  time  between  unscheduled  maintenance  actions 

mean  time  to  remove  and  replace  modules 

maintenance  training  support 

mean  time  to  shop  checkout 

mean  time  to  deliver 

mean  time  to  failure 

mean  time  to  repair 

mean  time  to  restore  function 

mean  time  to  restore  system 

mean  time  to  service 
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ABBREVIATIONS  (continued) 


MTTT 

mean  time  to  troubleshoot 

MTUWC 

mean  time  to  underwing  checkoutNDInondestructive  inspection 

NMC 

not  mission  capable 

NMCB 

not  mission  capable,  both  (i.e.,  maintenance  and  supply) 

NMCM 

not  mission  capable,  maintenance 

NMCS 

not  mission  capable,  supply 

NRTS 

not  reparable  this  station 

OA 

Directorate  of  Analysis,  AFOTEC 
obligation  authority 
operational  assessment 

OCM 

on-condition  maintenance 

OJCS 

Organization  of  Joint  Chiefs  of  Staff 

OL 

operating  location 

OMB 

Office  of  Management  and  Budget 

OPR 

office  of  primary  responsibility 

OPSEC 

operational  security 

OPTEVFOR 

Operational  Test  and  Evaluation  Force  (US  Navy) 

OR 

operationally  ready 

ORD 

operational  requirements  document 

ORLA 

optimum  repair  level  analysis 

O&S 

operational  and  support  (usually  used  in  relation  to  cost) 

O/S  CMP 

operational/support  configuration  management  procedures 

OSD 

Office  Secretary  of  Defense 

OT 

operating  time 

OT&E 

operational  test  and  evaluation 

OTEA 

Operational  Test  and  Evaluation  Agency  (US  Army) 

OU 

operational  unit  (sorties,  operating  hours,  flight  hours,  etc.) 

OUE 

operational  utility  evaluation 

P 

delay  time  in  preventive  maintenance 

PAD 

program  action  directive 

PAR 

program  assessment  review 

PCA 

physical  configuration  audit 

PCN 

production  control  number 

PCR 

program  change  request  or  publication  change  request 

PDM 

programmed  depot  maintenance 
program  decision  memorandum 

PDR 

preliminary  design  review 

PEM 

program  element  monitor 

PHT 

packaging,  handling,  and  transportation 

PID 

program  introduction  document 

PMC 

partially  mission  capable 

PMC8 

partially  mission  capable,  both  (i.e.,  both  maintenance  and  supply) 

PMCM 

partially  mission  capable,  maintenance 

PMCS 

partially  mission  capable,  supply 

PMD 

program  management  directive 

PME 

precision  measurement  equipment 

PMEL 

precision  measurement  equipment  laboratory 

PMP 

program  management  plan 

PO/ILSO 

program  office/integrated  logistics  support  office 

POL 

program  objective  memorandum 
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POM 

PPBS 

PRAT 

PSOC 

QAP 

QED 

Q-GERT 

QOT&E 

QPA 

QTAT 

R 

RAD 

RADC 

RAM 

RCMP 

RCS 

R&D 

RDGT 

RDT&E 

RFP 

RILSA 

RLA 

R&M 

RMMP 

RPIE 

RQT 

RT 

RTO 

RTOK 

SAC 

SAP 

SAR 

SATAF 

SBSS 

SCIR 

SCI 

SCP 

SD 

SDDM 

SE 

SECDEF 

SEDS 

SEM 

SETM 

SGR 

SIOP 

SISMS 

SLAM 

SM 

SMR 


ABBREVIATIONS  (continued) 


program  objective  memorandum 
planning,  programming,  budgeting  system 
production  reliability  acceptance  test 
preliminary  system  operation  concept 

questionnaire  analysis  program 
quantitative  evaluation  of  deficiencies 
queue-graphics  evaluation  review  technique 
qualification  operational  test  and  evaluation 
quantity  per  application 
quick  turnaround  time 

reliability 

reliability  analysis  center 
Rome  Air  Development  Center 
reliability,  availability,  and  maintainability 
reliability-centered  maintenance  program 
report  control  system 
research  and  development 
reliability  development/growth  testing 
research,  development,  test  and  evaluation 
request  for  proposal 

resident  integrated  logistics  support  activity 

repair-level  analysis 

reliability  and  maintainability 

reliability,  maintainability  management  plan 

real  property  installed  equipment 

reliability  qualification  tests 

recovery  time 

responsible  test  organizations 
retest  okay 

Strategic  Air  Command 
Secretary  of  the  Air  Force 
selected  acquisition  reports 
site  activation  task  force 
standard  base  supply  system 
subsystem  capability  impact  reporting 
system  command  language 
system  concept  paper 

Space  Division  (Air  Force  Systems  Command) 

Secretary  of  Defense  decision  memorandum 

support  equipment 

Secretary  of  Defense 

system  effectiveness  data  system 

software  evaluation  manager 

software  evaluation  team  member 

sortie  generation  rate 

single  integrated  operational  plan 

standard  integrated  support  management  system 

simulation  language  for  alternative  modeling 

system  manager 

source,  maintainability,  recoverability 
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SOC 

SOW 

SPO 

SPR 

SR 

SRD 

SRU 

ST 

ST/BIT 

TAC 

TAFT 

TAIDB 

TAT 

TCM 

TCTO 

TD 

TDR 

TDSD 

TDT 

T&E 

TE 

TEMP 

T/G 

T&HP 

TIP 

TMT 

TO 

T&IPR 

TOM 

TO  V&V 

TPG 

TPM 

TPO 

TPR 

TPWG 

TRANSEC 

TRD 

TS 

TSE 

TSTM 

TT 

UDL 

UR 

USDRE 
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ABBREVIATIONS  (continued) 


system  operational  concept 
statement  of  work 
system  program  office 
secretarial  program  review 
service  report 

standard  reporting  designator 
shop-replaceable  unit 
standby  time 
self-test/built-in  test 

Tactical  Air  Command 
test-analyze-fix  test 
tank-automative  integrated  data  base 
turnaround  time 

total  corrective  maintenance  time 
time  compliance  technical  order 
test  directive 
test  discrepancy 
teardown  deficiency  report 
training  device  support  data 
total  downtime  (TMT  &  ALDT) 
test  and  evaluation 

Directorate  of  Test  and  Evaluation,  AFOTEC 

test  and  evaluation  master  plan 

threshold  and  goals 

transportation  and  handling  provisions 

test  implementation  plan 

total  active  maintenance  time  (TCM  +  TPM) 

technical  order 

random  time 

technical  order  in  process  review 

test  operating  manual 

technical  order  validation  and  verification 

test  planning  group 

total  scheduled  maintenance  time 

test  program  outline 

training  program  requirements 

test  planning  working  group 

transmission  security 

test  requirements  document 

training  suitability 

top  secret 

training  supportability  evaluators 
training  supportability  test  manager 
total  time  (possessed  time  in  AFR  66-110) 

unit  detail  listing 
uptime  ratio 

Under  Secretary  of  Defense  for  Research  and  Engineering 


VETMIS 

VIDS/MAF 


vehicle  technical  management  system 

visual  information  display  system/maintenance  action  forms 
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ABBREVIATIONS  (continued) 


v&v 

validation  and  verification  (of  technical  orders) 
verification  and  validation  (of  software) 

WBS 

work  breakdown  structure 

WDC 

when-discovered  code 

WRM 

war  reserve  material 

WRSK 

war  readiness  spares  kit 

WRU 

weapon-replaceable  unit 

WSESA 

weapon  system  and  equipment  support  analysis 

WSR 

weapon  system  reliability 

WUC 

work  unit  code 

XP 

Directorate  of  Plans  and  Policy,  AFOTEC 
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DATA  ITEM  DESCRIPTION 


Form  Approved 
OMB  No.  070*41 33 


1  IDENTIFICATION  NUMBER 


SOFTWARE  MATURITY/RELIABILITY  DATA 


3.  DESCRIPTION /PURPOSE 

3.1  The  software  maturity/reliability  data  identifies  problems  detected  in  the  deliverable  software 
or  documentation  that  has  been  placed  under  contractor  configuration  control. 

3.2  The  software  maturity/reliability  data  are  used  by  HQ  AFOTEC  to  support  early  cpe's; 
assessments  (EOAs)  and  initial  operational  test  and  evaluation  (IOT&E). 


4.  APPROVAL  DATE 
(YYMMOO) 


5  OFFICE  OF  PRIMARY  RESPONSIBILITY  (OPR) 


64.  OTIC  REQUIRED  66.  GIDEP  REQUIRED 


7.  APPLICATION/ INTERRELATIONSHIP 


"  f  This  Data  Item  Description  (DID)  contains  the  format  and  content  preparation  instructions  ‘t- 
generated  under  the  work  task  described  by  paragraphs  4.1.9  and  4.1.10. 

7.2  The  Contract  Data  Requirements  List  should  specify  whether  these  data  are  to  be  precare'; 
celivered  on  bound  8  1/2  by  11  inch  bond  paper  or  electronic  media.  If  electronic  media  are  seiecte: 
precise  format  must  be  specified. 

(continued  on  ca 


APPROVAL  LIMITATION 


10.  PREPARATION  INSTRUCTIONS 


9a.  APPLICABLE  FORMS 


9b.  AMSC  NUMBER 


10.1  Content  and  format  instructions.  Production  of  these  data  using  automated  techniques  is  encc-rag-; 
Specific  content  and  format  instructions  for  these  data  are  identified  belowt- 

a.  Response  to  top  tailoring  instructions.  In  the  event  that  a  paragraph  or  subparagraph  -as  c~ 
tailored  out,  a  statement  to  that  effect  shall  be  added  directly  following  the  heading  cf  ea 
such  (sub)paragraph.  If  a  paragraph  and  all  of  its  subparagraphs  are  tailored  out.  . 
highest  level  paragraph  heading  need  be  included. 

b.  Use  of  alternate  presentation  styles.  Charts,  tables,  matrices,  or  other  presentation  sty  -as  ; 
acceptable  when  the  information  required  by  the  paragraphs  and  subparagraphs  of  :rs  : 
can  be  made  more  readable. 

c.  Format.  The  preferred  format  is  on  electronic  medium  as  specified  in  the  following  paragrac 

d.  Document  control  numbers.  For  hardcopy  formats,  this  document  may  be  printed  on  one 
both  sides  of  each  page  (single-sided/double-sided).  All  printed  pages  shall  contain  : 
document  control  number  and  the  date  of  the  document  centered  at  the  top  of  the  cage 

10.2  On  a  monthly  basis,  the  contractor  shall  supply  the  following  data  for  each  software  prcb:e'~ 
ennancement  in  the  specific  format  on  a  5  1/4"  floppy  disk,  IBM  PC  format,  or  VAX  VMS,  readable  9  :ra 
ape.  The  preferred  file  structure  is  DBase  111+  or  equivalent. 


(continued  on 


II.  DISTRIBUTION  STATEMENT 


DISTRIBUTION  STATEMENT  A.  Approved  for  public  release;  distribution  is  unlimited. 


JlIN  86 


Prevrovt  editions  ere  obsolete. 


PAGE  _ OF  _  PAGES 


,3-2 
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7.  APPL1CAT10N/INTERRELAT10NSHIP  (continued) 

7.3  Submission  of  software  maturity/reliability  data  will  begin  with  configuration  control  or  f'~ 
software  at  full-scale  development  (FSD). 

7.4  HQ  AFOTEC  will  use  the  data  and  send  results  to  the  system  program  office  (SPO)  for 
further  dissemination. 


10.  PREPARATION  INSTRUCTIONS  (continued) 


Description 

DBase  111+ 

Format 

lyge 

Lenath 

a. 

Software  Problem  Number 

Character 

10 

b. 

Description  of  Problem 

Character 

42 

c. 

CSCI  affected 

Character 

i  n 
•  ^ 

d. 

Priority  of  the  Problem  (1-5)  1  is  the  most  severe 
and  5  the  least.  IAW  2167A  Appendix  C 

Character 

e. 

Date  problem  discovered 

Date 

8 

f. 

Date  problem  was  fixed 

Date 

3 

g- 

Date  problem  was  closed 
(implemented  and  verified) 

Date 

8 

h. 

Enhancement/Fault  (E  or  F) 

Character 

1 

i. 

Number  of  lines  of  code  added  (executable  source  lines) 

Numeric 

6 

j- 

Number  of  lines  of  code  changed  (executable  source  lines) 

Numeric 

6 

k. 

Number  of  lines  of  code  deleted  (executable  source  lines) 

Numeric 

r\ 

o 

1. 

Number  of  people  it  took  to  make  the  change 

Numeric 

5 

1 .  Number  of  man-hours  it  took  for  skill  level  3  to 
make  changes. 

Numeric 

5 

2.  Number  of  man-hours  it  took  for  skill  level  5  to 
make  changes. 

Numeric 

5 

3.  Number  of  man-hours  it  took  for  skill  level  7  to 
make  changes. 

Numeric 

5 

4.  Number  of  man-hours  it  took  for  skill  level  9  to 
make  changes. 

Numeric 

5 

5.  Number  of  man-hours  for  engineers  to 
make  changes. 

Numeric 

5 

m. 

Number  of  occurrences  of  the  same  problem 

Numeric 

5 

n. 

Function  affected.  The  operational  function  of  the 
component  affected  by  the  trouble. 

Character 

MEMO 

0. 

Responsible  Module.  Component  to  which  programmer 
isolated  the  problem  and  complete  identification  of 
the  component,  version,  date,  and  any  other  significant 
component  identification  data. 

Character 

MEMO 

P- 

Test  Step.  The  test  procedure  and  step  being  executed 
at  the  time  the  trouble  was  discovered  (NA  for 
documentation  and  logic  troubles). 

Character 

MEMO 

q- 

Testing.  Describe  the  testing  performed  to  verify  the 
trouble  and  to  ensure  the  correctness  and  completeness  of 
the  change(s). 

Character 

MEMO 

10.3  On  a  monthly  basis,  the  contractor  shall  supply  the  following  day-to-day  system  failure 
data  in  the  specific  format  on  5  1/4"  floppy  disk,  IBM  PC  format,  or  VAX  VMS/readable  9  track 
tape. 


* 
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Description 


DBase  111+  Format 
Type  Length 


a.  Date  Date  3 

b.  Action  taken:  Character  10 

-  (sysjdle  -->  system  is  idle,  not  running  relevant 

programming) 

-  (sys_on  -->  system  turned  on) 

-  (sys_off  -->  system  powered  down  by  operation) 

-  (Software  problem  number) 

-  (Hardware  problem  number) 

-  (sys_sw  -->  running  system  software) 

c.  Time  of  action  (military  clock)  no  seconds  Numeric  4 


10.4  On  a  monthly  basis,  the  contractor  shall  supply  the  following  test  data  in  the  specific 
format  on  5  1/4"  floppy  disk,  IBM  PC  format,  or  VAX  VMS  readable  9  track  tape. 


Description 


a. 

b. 

c. 

d. 

e. 


Test  Identification  Number 
Test  Description 
Scheduled  Test  Date 
Completed  Test  Date 
Test  Status  (S  -  successful  test) 
(F  -  failed  test) 

(1  -  incomplete  test) 


DBase  111+  Format 

Type  Length 

Character  1 0 

Character  MEMO 
Date  8 

Date  3 

Character  1 


